Introduction
Survey experiments allow researchers to explicitly control the conditions of independent variables, circumventing endogeneity concerns in observational data and providing high internal validity. But how broadly can we learn from survey-experimental estimates, i.e., to what extent do they also exhibit external validity? Past literature has focused especially on whether estimates generalize to target populations. This is understandable, as population-representative sampling (as, e.g., recommended by Mutz Reference Mutz2011) is rarely used in practice, while estimates from nonprobability samples only allow for population-based inference under strong assumptions (Druckman Reference Druckman2022, Ch. 3).Footnote 1 External validity has multiple components, however (Druckman Reference Druckman2022; Egami and Hartman Reference Egami and Hartman2023): generalizability across populations, but also contexts, outcomes, and treatments.Footnote 2 In line with recent debates (e.g., Diaz, Grady, and Kuklinski Reference Diaz, Grady, Kuklinski, Curini and Franzese2020; Pepinsky Reference Pepinsky, Box-Steffensmeier, Christenson and Sinclair-Chapman2023; Samii Reference Samii, Box-Steffensmeier, Christenson and Sinclair-Chapman2024), I am concerned with the latter two: understanding whether survey-experimental evidence, even if it were population-representative, conforms to evidence as if derived from real-world variation (Sniderman Reference Sniderman2018) – which is pivotal for its relevance (Samii Reference Samii, Box-Steffensmeier, Christenson and Sinclair-Chapman2024). Mutz and Kim (Reference Mutz, Kim and Erin2020, 18) call this “the question of greatest importance to the future of survey experiments […] Do both real-world and survey-experimental treatments produce the same extent of effects when people are exposed?”
Incerti (Reference Incerti2020) suggests, via a meta-analysis, that survey-inherent biases and mismatches in stimulus provision between survey and field settings may complicate transportability. This parallels the difficulty of generalizing from laboratory experiments to a real world where multiple treatments co-occur, stakes are higher, and behavior has meaningful consequences (Levitt and List Reference Levitt and List2007). Controlled, within-study comparisons are scarce, though, due to demanding conditions: having exogenous variation in the first place (trivial for survey experiments, but challenging for real-world data), while holding other elements of external validity (e.g., study population, temporal context) constant between evidence types, and while studying treatments and outcomes which are reasonably comparable across both data sources. Consequently, the small literature contrasting survey-experimental and observational estimates paints a mixed picture of external validity. Some works find conformity to varying degrees (e.g. Auerbach and Thachil Reference Auerbach and Thachil2018; Hainmueller, Hangartner, and Yamamoto Reference Hainmueller, Hangartner and Yamamoto2015; Petzold and Wolbring Reference Petzold and Wolbring2018; Rudolph, Freitag, and Thurner Reference Rudolph, Freitag and Thurner2024), while others do not (e.g. Findley et al. Reference Findley, Laney, Nielson and Sharman2017; Whiting and Ma Reference Whiting and Ma2021) – which might be due to limits to generalizability, or issues with within-study comparability in the first place.
In this article, I present new evidence, focusing on vignette survey experiments and generalizability across treatments and outcomes. I exploit a unique setting where comparability seems high ex ante: with close temporal proximity and for the same sample, I study migration intentions and actual migration under exposure to natural disasters in both a survey- and a quasi-experimental design. Specifically, I draw on a population-representative panel study of around 1,600 household heads in rural Bangladesh, living along the Jamuna River. At baseline, all households are at risk of riverbank erosion and flooding. Exploiting the quasi-random nature of erosion processes, I show that post-monsoon quasi-experimental estimates of actual erosion exposure align well with pre-monsoon survey-experimental estimates of hypothetical exposure, particularly for behavioral intent, and less for actual behavior.
In doing so, I draw on a case where actual exposure has relevant real-world consequences, and where it is of core interest to policymakers whether we can actually learn from hypothetical survey experiments. To my knowledge, I also provide the first evidence that distinguishes generalizability for behavioral intent from that for actual behavior.
Generalizing across treatments and outcomes
Factorial survey experiments have various desirable properties, which should all enhance generalizability: for example, they are seemingly robust to researcher demand effects (Mummolo and Peterson Reference Mummolo and Peterson2019) or the extent of hypotheticality (Brutger et al. Reference Brutger, Kertzer, Renshon, Tingley and Weiss2023), can mitigate social desirability bias (Auspurg et al. Reference Auspurg, Hinz, Liebig, Sauer, Jann, Lynn, Scherpenzeel and Patrick2014), and regularly yield consistent results across a variety of contextual conditions at the individual (Coppock, Leeper, and Mullinix Reference Coppock, Leeper and Mullinix2018) or country level (Bassan-Nygate et al. Reference Bassan-Nygate, Renshon, Weeks and Weiss2025). At the same time, for within-study comparisons Treischl and Wolbring (Reference Treischl and Wolbring2022, 159) find that “the current state of research is small and inconclusive about the predictive validity of [factorial survey experiments comparing] hypothetical decision-making and a behavioural benchmark” – where some studies find non-conforming results; some conforming results; others conforming marginal effects with varying baseline distributions. This assessment mirrors my overview in Appendix Section A.1 on the small literature that tests the generalizability of survey-experimental treatments for outcomes of interest to political science.
Nonconforming results could stem from three sources, however: first, conceptual mismatches regarding measures of treatment or outcome where we can expect at most sign-generalization (Egami and Hartman Reference Egami and Hartman2023) in the first place; second, complications from compound differences between survey and real-world settings, where multiple aspects – populations, contexts, treatments, and outcomes – may jointly differ; and third, a genuine lack of external validity of survey-experimental treatments or outcomes, given “fabricated settings” (Samii Reference Samii, Box-Steffensmeier, Christenson and Sinclair-Chapman2024, 13) in surveys, where pretreatment information, expression of moderators, or heterogeneity in treatment interpretation differs from the real-world analogue (Diaz, Grady, and Kuklinski Reference Diaz, Grady, Kuklinski, Curini and Franzese2020). In that case, survey-experimental evidence consequently fails to inform real-world microfoundations of social science theory (Pepinsky Reference Pepinsky, Box-Steffensmeier, Christenson and Sinclair-Chapman2023). I address the first two sources of nonconformity by design, such that any remaining divergence between survey-experimental and quasi-experimental estimates is attributable to genuine limits to external validity rather than confounds.
Regarding the second, in a best-case scenario, we would be able to construct a simultaneous two-arm experiment for the same population where only treatment mode and outcome measurement (survey vs. real-world) differ. While I cannot exploit such a design, I can draw on a second-best alternative: a panel-study setup (i.e., identical sample and geography) with temporally proximate survey-experimental and quasi-experimental exposure, which allows me to hold population and context nearly constant.
Regarding the first, I propose that general design principles for survey experiments – constructing treatments that exhibit realism while drawing on outcome measures that capture the real-world phenomena of interest (Huber and Graham Reference Huber, Graham, Snowberg and Yariv2025) – also inform the comparability of survey-experimental estimates with real-world estimates. Concerning treatment realism, we have to distinguish “experimental” from “mundane realism” (Druckman Reference Druckman2022, Ch. 3). “Mundane realism” implies a treatment that respondents regularly encounter in their everyday lives. This is not necessary for testing social science theories – which only require underlying constructs to be successfully manipulated (Druckman Reference Druckman2022; Mutz Reference Mutz, Druckman and Green2021). But for the purpose of directly comparing survey- and real-world treatments within a single study, “mundane realism” is actually a desirable property, as it allows one to assess both effect- and sign-generalization (Egami and Hartman Reference Egami and Hartman2023). My vignette experimental treatment, communicating hypothetical exposure to natural disasters among a population at risk, captures such “mundane realism” – albeit in stylized form.
Regarding outcome measures, an ideal survey experiment mirrors the specificity and stakes of the real world (Huber and Graham Reference Huber, Graham, Snowberg and Yariv2025). I can address the specificity requirement directly by comparing hypothetical to actual behavioral intent (to migrate) with identically worded survey questions, and by using a survey outcome very closely aligned with the behavioral outcome (migration as measured through household location). Mirroring the stakes is challenging, given that choices are inherently hypothetical in survey experiments (Mutz Reference Mutz, Druckman and Green2021). Fortunately, I can compare survey-experimental estimates of hypothetical intent to both real-world behavioral intent (where stakes are similarly low) and actual behavior (where stakes obviously differ greatly).
In sum, both inquiries investigate the very same population and spatial context in a temporally proximate comparison. This allows me to focus on generalizability across treatments and outcomes – while drawing on a treatment that aligns with respondents’ livelihoods and real-world experiences, and assessing realistic, tangible outcomes using identical measures for stated and actual behavioral intent, as well as an intent measure closely aligned with actual behavior.
Research design
The data (Freihardt, Rudolph, and Koubi Reference Freihardt, Rudolph and Koubi2026; Rudolph Reference Rudolph2026) stem from multiple waves of high-quality, face-to-face and phone surveys in 2021 based on a geospatial random draw of 1,684 river-proximate household heads residing in 36 locations distributed along 250 km of the eastern bank of the Jamuna River in Bangladesh, where all households are at real-world risk of riverbank erosion and flooding (Freihardt, Rudolph, and Koubi Reference Freihardt, Rudolph and Koubi2021). Figure 1 gives an overview of sampling.
Overview of the Jamuna River, the 36 study locations, and three zones for geospatial random sampling within locations. Locations comprise one-kilometer stretches along the eastern bank unprotected from erosion. Copyright: map: Google; satellite images: TerraMetrics, 2022. Reproduced from Rudolph, Koubi, and Freihardt (Reference Rudolph, Koubi and Freihardt2025).
Figure 1. Long description
The image consists of three main elements: a regional map of Bangladesh, a zoomed-in map of the Jamuna River, and a detailed satellite image of one study location. The regional map shows the geographical context, highlighting the Jamuna River and the 36 study locations along its eastern bank. The zoomed-in map provides a closer view of the Jamuna River, marking the study locations with red dots. The satellite image offers a detailed view of one specific study location, showing a one-kilometer stretch along the eastern bank unprotected from erosion. This area is divided into three zones for geospatial random sampling, marked with different colored dots. The satellite image includes measurements of 50 meters and 100 meters, indicating the scale and layout of the sampling zones.
A pre-monsoon survey contained an experimental vignette communicating hypothetical damage and risk from riverbank erosion and floods in a 3 × 2 design, with migration intent as the outcome. Post-monsoon follow-up phone surveys with the same sample captured real-world affectedness one and two months later, with household heads’ real-world migration intentions and observed migration as outcomes.Footnote 3 The survey also inquired about numerous other respondent characteristics (demographics, income, and a broad array of perceptions and attitudes), as part of a larger project.
As described in Appendix B.1, the strategy for obtaining survey-experimental and real-world exposure was pre-registered, as was the geospatial sampling strategy. The conceptual question investigated in this article was developed ex post. As this leaves potentially problematic leeway, e.g., regarding consequential treatment operationalization decisions (Quoß and Rudolph Reference Quoß and Rudolph2026), I used experimental conditions and outcome measures as pre-registered or as applied by previously published works with the same data to reduce researcher degrees of freedom. Appendix B.2 provides for JEPS reporting standards for experimental research.
Case context
Bangladesh’s low-lying deltaic geography makes it highly exposed to climate-related hazards (IPCC Reference Masson-Delmotte, Zhai, Piran, Connors, Péan, Berger, Caud, Chen, Goldfarb, Gomis, Huang, Leitzell, Lonnoy, Matthews, Maycock, Waterfield, Yelekç, Yu and Zhou2021). Riverbank erosion constitutes one of the most spatially concentrated and economically damaging processes in this context (Ahmed Reference Ahmed2015; Kumari Rigaud et al. Reference Kumari Rigaud, Sherbinin, Jones, Bergmann, Clement, Ober, Schewe, Adamo, McCusker, Heuser and Midgley2018). Erosion occurs particularly along the Jamuna River, one of the world’s most dynamic braided river systems, experiencing substantial and uneven erosion driven by upstream hydrological and sediment dynamics (CEGIS 2018; Oberhagemann, Haque, and Thompson Reference Oberhagemann, Haque and Thompson2020). Protection against riverbank erosion can occur naturally through riverine islands (so-called Chars) or man-made structures (e.g., sandbags). Sampling site selection excluded protected communities – among unprotected communities, the timing and location of severe erosion events are largely unpredictable (Freihardt and Frey Reference Freihardt and Frey2023), particularly at the household level (Freihardt Reference Freihardt2024). Consequently, affectedness is plausibly orthogonal to most observable socioeconomic characteristics (see Appendix Figures A.5 and A.6). Hence, I treat erosion along the Jamuna as a likely exogenous shock, noting that erosion clusters at the location level.
The survey-experimental treatment
With the vignette, I chose to communicate two attributes to respondents: hypothetical household-level damage and future risk of affectedness in a 3 × 2 design, uniformly randomized.Footnote 4 Specifically, the vignettes detailed no, medium, or high impact on household assets. Regarding future risk, I used combinations of hypothetical statements about village elders as trustworthy elites, hypothetical (but plausible) government intervention, and village-level effects to construct credible variants of future risk (low/high). Table 1 provides an overview of the six vignettes and wordings.
Vignette overview (six versions, from three damage times two risk attribute levels, uniformly randomized) with realized sample allocation in brackets. Appendix Section A.2.1 provides exact wordings
Table 1. Long description
The table presents six vignettes detailing different levels of household damage and future risk. Each vignette is labeled from T0 to T5, with corresponding damage and risk descriptions. The damage levels range from no damage to high damage, while the risk levels are categorized as low or high. The sample allocation percentages for each vignette are also provided, ranging from 15.2 percent to 17.0 percent. The table includes specific descriptions of the damage and risk for each vignette, such as ‘no damage’ and ‘low risk’ for T0, and ‘high damage’ and ‘high risk’ for T5.
Given that households are continuously at risk of actual affectedness in this context, and given that the monsoon season was just about to begin, I consider these vignettes both highly realistic and relatable for respondents. I assess the plausibility of such “mundane realism” by a manipulation check on whether respondents report being able to relate to the vignettes – a core assumption for comparing them directly with real-world damage and risk. Ideally, I would find that all vignettes are perceived as equally plausible, while vignette texts do not, under any condition, affect real-world risk assessments (preventing spill-overs to actual behavior). To this end, I fielded two items capturing how realistic the vignette is perceived to be and how respondents assess the risk of real-world affectedness. Appendix Section A.2.2 provides the corresponding evidence. Vignettes are assessed as equally likely and do not systematically affect self-assessed real-world risk, though high-risk vignettes are somewhat associated with higher recurrence perceptions. Overall, I am confident I can compare hypothetical vignette replies to actual migration intentions/behavior, while holding realism constant.
The real-world treatment
Survey setting and natural disaster impact timing
To capture real-world affectedness and migration intentions/behavior, I primarily rely on phone surveys conducted one and two months after the vignette experiment on the identical study population. Attrition was low,Footnote 5 making me confident that I report results for the same underlying population – this is confirmed by equivalence tests (see Appendix Section A.2.5), indicating only minor imbalance among core observable socio-demographic, geospatial, and attitudinal variables.Footnote 6 Between the in-person (wave 1) and the later phone survey waves, the monsoon brought erosion and flooding that affected some, but not all, study participants.
As Appendix Figure A.3 shows, wave 2 is particularly suitable for comparison to the vignette. By this survey wave, erosion had just occurred (for about 50%) and had already affected households (for about 25% of respondents), meaning a substantial subset of the sample had been treated and could be surveyed in temporal proximity to that exposure. Moreover, floods had not yet occurred by wave 2, allowing me to focus on erosion as the natural phenomenon with the clearest quasi-experimental properties. As a caveat to this temporal structure, the monsoon was still ongoing in survey wave 2, making it possible that respondents did not interpret the treatment as a completed event (and hence different from the vignette communication) – which is why I also report comparisons to survey wave 3.Footnote 7
Independent variables for real-world affectedness
Regarding affectedness, the project inquired about occurrence (binary yes/no and extent) in the village, personal affectedness (binary), and type of personal affectedness (categorical). Appendix Section A.2.3 provides exact survey question wordings. From these variables, I construct the following measures to assess household-level and village-level affectedness (where I use the latter as a proxy for risk): First, I code a binary indicator of any affectedness, separately for erosion and floods. Second, I code an ordinal indicator for the severity of affectedness, again for both erosion and floods, building on the typology used in Rudolph, Koubi, and Freihardt (Reference Rudolph, Koubi and Freihardt2025). They differentiate strong, medium, other, and no impact.Footnote 8 I propose that the top two groups most plausibly correspond to the impact categories communicated in the vignette (high and medium damage). Third, as an indicator for risk, I code the share of other households within a respondent’s village affected, and dichotomize this measure at more than half affected versus fewer than half. This directly links to the risk communication in the vignette, which explained risk, among other things, by the extent of this year’s affectedness within the village – note that the vignette component drawing on statements by village elders and government action is not captured by the real-world risk proxy, making the risk aspect more difficult to interpret in comparison to the vignette, i.e., more akin to a comparison on the dimension of “experimental realism.”
Dependent variables: migration intentions and behavior
Pre-monsoon migration intentions in the vignette experiment
Following the vignette, the survey elicited several dependent variables; here, I focus on hypothetical movement behavior:
-
– Migration (move/stay/depends (unprompted)): “If you were in this situation, would you stay here in your present village or would you prefer to move to another place?”
-
– IF depends: “Now, if you really had to choose, would you stay in your present home or move somewhere else?”
These two survey questions allow me to construct my main outcomes of interest for the vignette experiment. I code two hypothetical migration intention variables, the first in a “narrow” (“1” if respondents indicate “move” to the first or second question), the second in a “broad” sense (“1” if “move” or “depends” in the first question).
Actual migration intentions and behavior in the post-monsoon surveys
To capture actual migration/intentions, I rely on three survey components. The first directly mirrors the dependent variable in the vignette experiment (and is coded identically), drawing on a non-experimental question on general migration aspirations:
-
– Migration (move/stay/depends (unprompted)): “Right now, if you could choose, would you stay here in this village or would you prefer to move to another place?”
-
– IF depends: “Now, if you really had to choose, would you stay in your present home or move somewhere else?”
A caveat to comparability is, of course, the temporal lag between exposure and outcome measurement in the real-world data (compared to immediacy in the vignette). To alleviate this concern, I also draw on a second measure of actual intentions available from the survey. This captures migration intentions over the preceding month, which likely reflects the actual impact (“yes” coded 1, else 0).
-
– “During the last month, have you thought seriously about leaving this village?” (yes/no/don’t know).
Regarding actual migration, I use information on actual behavior, obtained by asking about the household head’s current location at the start of the survey.
-
– “Do you currently live in the same village as when we spoke last time, meaning in [village name], or have you moved to another location?” (same/another [name])
Respondents are coded as “migrant” (1; 0 otherwise) if they state another location as their home village. Enumerators were instructed to ensure that respondents correctly understand “currently living,” i.e., exclude outside stays such as “visiting”/“traveling.”
Estimation
The vignette experiment relies on researcher-induced variation, allowing me to proceed directly to causal tests of how vignette attribute levels influenced responses. In supplementary models, I control for village and sampling zone, as this can increase the efficiency of estimation (Mutz Reference Mutz2011) and to mirror the estimation of real-world intentions/behavior.
Regarding actual affectedness, I assume quasi-random exposure. My approach is based on a carefully pre-registered design within a larger project which “ensured that our sample is representative of all households at risk of exposure to floods and riverbank erosion before the 2021 monsoon onset along the easternmost riverbank line of the Jamuna River” (Rudolph, Koubi, and Freihardt Reference Rudolph, Koubi and Freihardt2025), while this risk only materialized for some households ex post (see Appendix Section B). I am confident this strategy also properly ensures quasi-random exposure, at least once I take the location along the Jamuna and the sampling zone into account.
Both Rudolph, Koubi, and Freihardt (Reference Rudolph, Koubi and Freihardt2025) and Freihardt (Reference Freihardt2025) substantiate this assumption, showing that raw effects and effects after balancing for potential endogeneity yield highly comparable findings for the link between affectedness and migration/migration intentions. In addition, Appendix Figure A.4 shows that, with isolated exceptions for erosion in wave 2, neither actual flood occurrence nor erosion occurrence in waves 2/3/4 is linked to respondents’ likelihood assessments of flood/erosion occurrence in wave 1, once I include simple controls for location and sampling zone; the same is true for households’ erosion/flood affectedness.Footnote 9 Moreover, Appendix Figures A.5 and A.6 present equivalence tests (Hartman and Hidalgo Reference Hartman and Hidalgo2018), indicating treatment and control samples are equivalent on core observable socio-demographic/attitudinal characteristics. At the 5% level, I reject equivalence only for spatial indicators which I later control for (zone > 100m and geographical location) and, barely, for assets in wave 2 (p = 0.053).
Note that I follow Rudolph, Koubi, and Freihardt (Reference Rudolph, Koubi and Freihardt2025) and “impute” migration intentions when respondents migrate: intentions to migrate from the wave 1 household location can obviously not be measured when households migrate. This would lead to selective attrition that would bias any estimates. To circumvent this issue, I set all actual intentions to “1” for migrants. Note that the project inquired about erosion/flood affectedness identically in the non-migrant and migrant samples, specifically asking about erosion/flood occurrence at their prior place of residence for the latter.
For both vignette and real-world outcomes, I use linear regression with standard errors clustered by village, i.e., the level of real-world treatment assignment (Abadie et al. Reference Abadie, Athey, Imbens and Wooldridge2023), to estimate effects. To statistically compare estimates obtained from the vignette and the real-world data, I conduct z-tests based on seemingly unrelated estimates (SUESTs) for the joint sample (Clogg, Petkova, and Haritou Reference Clogg, Petkova and Haritou1995), i.e., respondents who answered the vignette and were present in wave 2/3. I apply the same regression specification, again clustering standard errors by location in the SUEST equation (Weesie Reference Weesie1999).
Results
Vignette experiment
I first present results for the vignette experiment. Table 2 presents treatment effects for hypothetical migration in the “broad” or “narrow” intent conceptualization, across vignettes with varying levels of risk and damage. The basic interpretation is straightforward: Both the communication of higher risk and of higher damage increase migration intent (irrespective of measure, albeit with differing statistical precision). Note that I interpret models 1 and 3 (without controls) below, given minimal differences between models with and without controls. Appendix Figure A.9 presents marginal means on hypothetical migration in the “broad” or “narrow” conceptualization to display overall migration intentions on an absolute scale.
Estimates from linear regressions of hypothetical narrow/broad migration intentions (see model header) on vignette attributes “risk” (low/high) and “damage” (none/medium/high). Where indicated, models include location and zone controls. Standard errors (in parentheses) clustered by location. Ref. indicates reference category.
Table 2. Long description
The table presents OLS regressions of migration intentions on vignette attributes of risk and damage. It includes four columns labeled Migration intent broad and Migration intent narrow, each with two sub-columns. The rows are categorized by Erosion/flood damage and Erosion/flood risk, with subcategories such as none, medium, high, and low. Each cell contains regression coefficients with standard errors in parentheses. The table compares the impact of different levels of risk and damage on migration intent, both broad and narrow. Location and zone controls are included where indicated. The data shows that higher risk and higher damage increase migration intent, with varying statistical precision.
+ p < 0.1, * p < 0.05, ** p < 0.01.
Control group (no damage/risk scenario, i.e., the constant) migration intention probabilities, even absent affectedness and risk, are meaningfully above zero: at around 16 percentage points (broad) and 11 percentage points (narrow). Medium damage increases narrow (by around 6 percentage points) and broad intentions (by around 4 percentage points, significant only at the 10% level). High risk of future damage increases narrow (by around 8 percentage points) and broad intentions (by around 12 percentage points). High damage increases narrow and broad intentions (by around 14 percentage points). This effect is significantly larger compared to communicating medium damage (p < 0.01 for both outcomes). Appendix Table A.1 provides interaction effects between risk and damage, indicating no clear pattern of interaction – estimates are mostly around zero, with the potential exception of a positive interaction between medium damage and risk for narrow migration intent, though not measured precisely.
For the later comparison to real-world behavioral intent/behavior, I derive the following expectations: Expectation 1: a baseline migration intention, controlling for affectedness and risk, of around 11–16 percentage points. Expectation 2: a moderate increase in migration intentions/migration with medium damage (around 4–6 percentage points), controlling for village-level affectedness (risk). Expectation 3: a significant and positive effect of village-level affectedness (risk) of around 8–12 percentage points, controlling for personal affectedness. Expectation 4: a significant and positive effect of strong erosion exposure of around 14 percentage points, controlling for village-level affectedness (risk). Expectation 5: a significantly larger effect of high damage compared to medium damage on migration and migration intentions, controlling for risk. Expectation 6: no clear pattern of interaction effects between risk and damage.
Actual migration intentions and migration behavior
Next, I turn to actual affectedness and actual migration intentions/behavior. The impact data I report on first stem from wave 2, where measurement of outcomes is temporally proximate to both exposure and the vignette. Also, given flood affectedness was negligible in wave 2 (see Appendix Figure A.3), this allows me to focus on the more cleanly identified erosion affectedness (while controlling for flood impact to address potential endogeneity concerns).
Table 3 reports results. Models 1, 5, 9, and 13 show that any erosion affectedness (averaging over all impact extent categories) is significantly and substantially related to both migration intentions – irrespective of past (model 1) or current narrow (model 9)/broad (model 5) measurement, by around 22 percentage points – and to actual migration (12 percentage points, model 13). This already indicates that affectedness relates to hypothetical intent, actual intent, and, in a dampened way, actual behavior. These estimates also clearly indicate sign-generalization across multiple outcome measures in both the vignette and the observational data. When comparing coefficient effect sizes statistically to high damage in the vignette, I find no evidence for statistical differences at the 5% level (see Appendix Table A.3, but note that p broad intent = 0.053).
Wave 2 estimates from linear regressions of actual migration intentions/behavior (see model header) on erosion damage (any reported impact/impact extent) and erosion risk (village-level affectedness), depending on model specification. Models control for flood impact and include location/sampling zone fixed effects. Standard errors (in parentheses) clustered by location. Reference category for erosion/flood damage: none reported; for risk: below half of households (hhs) report erosion affectedness.
Table 3. Long description
The table presents results from models 1, 5, 9, and 13, illustrating the relationship between erosion affectedness and migration intentions. Erosion affectedness significantly influences both migration intentions and actual migration, with around a 22 percentage point impact on intentions and a 12 percentage point impact on actual migration. The data indicates that affectedness relates to hypothetical intent, actual intent, and actual behavior. The estimates show sign-generalization across multiple outcome measures in both vignette and observational data. The table includes standard errors clustered by location and controls for flood impact and location/sampling zone fixed effects. Key findings are highlighted with statistical significance and substantial effect sizes.
+ p < 0.1, * p < 0.05, ** p < 0.01.
For maximum comparability, I now turn to the concrete expectations derived from the vignette. Regarding Expectation 1, baseline hypothetical migration rates (vignette control condition, 11–16 percentage points) conform to baseline migration intention rates (see constants in Appendix Table A.2), which are between 6 percentage points and 20 percentage points (depending on measure, most proximate are past migration intentions with 16 percentage points). They are substantively different for actual migration with 1 percentage point. A direct comparison of estimates confirms this: as indicated in Appendix Table A.4, I observe a significant difference between the hypothetical intent measure and actual migration, but not actual intent.
Expectation 2 concerned medium damage, controlling for future risk (4–6 percentage points). Comparing to Table 3, models 3, 7, 11, 15, which control for village-affectedness (risk), I find increases between 8 to 17 percentage points for the actual intention measures, and 2 percentage points (insignificant) for actual migration. Hence, medium damage has effects on actual intentions that are similar to or notably higher than vignette estimates, depending on the measure, and may be underestimated for actual migration behavior. However, the evidence does not allow for strong conclusions. The former differences are only partly statistically significant (p broad intent = 0.015, p narrow intent = 0.818), the latter are not (p migration = 0.103) (see Appendix Table A.5).
Expectation 3 concerned no damage, but high future risk (8–12 percentage points). As evident from models 3, 7, 11, and 15, risk measured by high village-level affectedness, controlling for household affectedness, is likewise positively related to migration intentions and migration – by 12 to 13 percentage points for past and broad current intent, and 10 percentage points for narrow current intent. These results are highly in line with the vignette. The 2 percentage point estimate for actual migration (insignificant) falls below the narrow hypothetical measure from the vignette. Note that none of these differences are statistically significant at the 5% level (but note that p migration = 0.077) (see Appendix Table A.6).
Expectation 4 concerned high damage, controlling for future risk (around 14 percentage points). Models 3, 7, 11, and 15 show that, controlling for risk, high damage increases past (by 22 percentage points), narrow (37 percentage points), and broad (29 percentage points) intentions, and also actual migration (30 percentage points) – indicating a stronger reaction in behavioral intent as well as actual behavior compared to the vignette. These differences are all statistically significant (see Appendix Table A.7).
Expectation 5 indicated that respondent reactions to high damage were larger than reactions to medium damage (around 8 to 10 percentage points in the vignette). Comparing the coefficients for high vs. medium damage (models 3, 7, 11, and 15), I confirm this in all regressions. Notably, the difference is significantly larger in the observational compared to the survey-experimental data for actual migration and the narrow actual intent measures, while no difference is discernible for broad actual intent (see Appendix Table A.8).
Last, I explore interactive models (4, 8, 12, 16), crossing households’ affectedness with village-affectedness (risk). The vignette experiment indicated mostly null effects in this regard, except for a positive relation between medium damage and risk in one model (insignificant at conventional levels) (Expectation 6). In the observational data, models 12 and 16 show that the interaction yields both a positive and substantively relevant coefficient for household affectedness and narrow intent (by 11 percentage points) and actual migration (5 percentage points) – moreover, additional exposure to risk increases narrow intent even further (18 percentage points), as well as actual migration (14 percentage points). The latter effect is only significant at the 10% level, however. For past intent and broad current intent, risk and household affectedness have their own, sizable, independent effects, but no cumulative effects (see models 4 and 8), more in line with the vignette.
Taken together, I find broad evidence for sign-generalization regarding the overall effect of damage in vignette and observational data. In addition, regarding expectations 1–6, effect-generalization for hypothetical intent is plausible for 1) baseline behavioral intent (but not behavior), 2) medium damage, and 3) risk – while for both 2) and 3) coefficients for actual behavior are underestimated. Regarding 4) and 5), high damage and the increase in effect size with high compared to medium damage, I find substantially larger, and mostly significant, estimates in the observational data. Finally, regarding 6), the interaction between risk and affectedness, I find an inconclusive picture across both datasets.
Robustness
While the abovementioned comparisons have the great advantage of relying on the temporally most proximate measure available to me, I assess robustness for three reasons. First, theoretically, potential conceptual mismatches regarding the treatment variables might hamper comparability. On the one hand, the wave 2 real-world affectedness stems predominantly from erosion, while in drafting realistic vignette scenarios I relied on specific damage scenarios from both erosion and floods. On the other hand, affectedness was communicated as concluded in the vignette, while erosion was an ongoing process in wave 2 (largely concluded by wave 3) – hence, we might worry that an ongoing erosion process makes for a different interpretation of affectedness in wave 2, both regarding damage (having the potential to increase) and the risk variable (signaling danger of being affected still this, and not next monsoon season, as in the vignette). Second, later data alleviate potential conceptual mismatches for the dependent variable, actual migration, as actual behavior might just take longer to materialize. Third, variation in affectedness by later waves yields additional independent operationalization of disaster affectedness, alleviating concerns for spurious results (Quoß and Rudolph Reference Quoß and Rudolph2026).
Comparison to affectedness after 2 months
When comparing the vignette estimates to the patterns observed for wave 3 (see Table 4), a broadly consistent picture emerges. Overall, effect sizes are attenuated compared to wave 2 (see Table 3), potentially due to a greater average temporal distance between treatment and outcome measure. Still, models 1, 5, 9, and 13 show that erosion impact is consistently and significantly linked to actual intent and behavior, and as models 2, 6, 10, and 14 show, this is more strongly the case for strong exposure. As evident from models 3, 7, 11, and 15, the risk variable only relates to the past and broad intent measures, not to narrow intent and actual migration. This pattern strongly underpins sign-generalization for all outcomes regarding damage; regarding risk, this is only partly the case for intent measures, and not for actual behavior, however. Regarding effect-generalization, and relating to the concrete expectations from the vignette, I note the following differences significant at the 5% level (see Appendix Tables A.11 to A.16). Baseline migration (intent) is lower for all actual measures. Risk is estimated lower (null) for narrow intent and actual behavior, but not broad intent. The high damage estimate is larger for narrow intent, but not broad intent and actual behavior. The high–medium damage difference is larger for actual behavior, but not the intent measures. Finally, similar to the vignette, I find no interaction effects between risk and affectedness for the actual intent measures – however, a significantly positive interaction emerges for actual behavior.
Wave 3 estimates from linear regressions of actual migration intentions/behavior (see model header) on cumulative wave 2/3 erosion damage (any reported impact/impact extent) and erosion risk (village-level affectedness), depending on model specification. Models also include cumulative wave 2/3 flood damage (any reported impact/impact extent) and flood risk (village-level affectedness). Models include location/sampling zone fixed effects. Standard errors (in parentheses) clustered by location. Reference category for erosion/flood damage: none reported; for erosion/flood risk: below half of households (hhs) report erosion/flood affectedness.
Table 4. Long description
The table presents OLS estimates of the cumulative impact of erosion and flood on migration aspirations and actual migration. It includes various models with different specifications, showing the effects of erosion and flood impact on past, current broad, current narrow, and actual migration aspirations. The table has 16 columns and multiple rows, with each row representing different models and their respective estimates. Key trends include significant links between erosion impact and actual migration intent and behavior, with stronger effects for high exposure. The risk variable relates to past and broad intent measures but not to narrow intent and actual migration. Interaction effects between risk and affectedness are noted for actual behavior but not for intent measures.
+ p < 0.1, * p < 0.05, ** p < 0.01.
Turning to the estimates of flood damage/risk, I find sign-generalization for past and broad actual intent, but not for narrow actual intent and actual behavior.
Taken together, I find strong conformity between the vignette estimates and the past/broad actual intent measure. Core results for narrow intent/actual behavior also replicate with respect to sign (for the damage treatment). Effect-generalization is low for actual behavior estimates, however.
Comparison to affectedness after 6 months
As a last check, given that intent, and especially actual migration, may take longer to materialize, I compare vignette results to results for actual intent/behavior obtained by Rudolph, Koubi, and Freihardt (Reference Rudolph, Koubi and Freihardt2025) and Freihardt (Reference Freihardt2025) with the same data, six months after the baseline in a follow-up in-person survey – hence medium term impacts of cumulative affectedness during the full monsoon season. Rudolph, Koubi, and Freihardt (Reference Rudolph, Koubi and Freihardt2025) report increases in actual migration intentions with high erosion impact extent comparable to the vignette (see their Tables 1 and 2, coefficients around 15–25 percentage points depending on intent measure). Likewise, strong village-level affectedness (risk) controlling for households’ affectedness increases intentions by 15–23 percentage points (see their Appendix Tables B16 and B17, depending on intent measure). Freihardt (Reference Freihardt2025) reports corresponding results for actual migration (increase from affectedness by 18–22 percentage points (see his Table S11)), as well as a positive relation between village-level affectedness and actual migration (see his Table S16). Again, these patterns are clearly indicative of sign-generalization.
Conclusion
Do survey experiments generalize to real-world effects? I study this question using a population-representative panel study of around 1,600 household heads in rural Bangladesh, living along the Jamuna River, where riverbank erosion and flooding pose a recurring threat. Exploiting close temporal proximity between a pre-monsoon vignette experiment on hypothetical disaster exposure and risk, as well as post-monsoon quasi-experimental actual exposure for the same population, I compare the survey-experimental and real-world relation between disaster affectedness/risk and migration intent/behavior. Within-study comparisons of this type are rare (Treischl and Wolbring Reference Treischl and Wolbring2022), and when conducted, they are usually complicated by the joint variation of multiple elements of external validity (Egami and Hartman Reference Egami and Hartman2023). In my case, however, population and context are held nearly constant between evidence types, allowing me to focus specifically on generalization across treatments and outcomes.
I find sign-generalization across all outcome types: both hypothetical and actual affectedness and risk significantly increase migration intentions, and this extends to actual migration behavior for the damage treatment. The vignette performs particularly well in replicating baseline migration intent, the effects of medium damage, and risk, where results are consistent with effect generalization. At the same time, the vignette likely underestimates migration intent and behavior in high-damage scenarios, where real-world reactions are substantially stronger – suggesting that the grave consequences of personal affectedness may not be fully conveyed by textual descriptions. Interactive effects in the observational data also do not correspond to affectedness-risk interactions in the survey-experimental scenarios. In addition, I also observe sign-generalization for both short- (one/two months) and medium-term (six months) outcomes, which is encouraging given the usual difficulty survey experiments face in arguing for a real-world persistence of effects (Huber and Graham Reference Huber, Graham, Snowberg and Yariv2025). Overall, vignette-experimental results conform more closely to actual behavioral intent using proximate measures.
These results are good news for scholars who plan to research (migration) behavior with (factorial) survey experiments. My case contrasts with non-conforming results in comparable studies (Incerti Reference Incerti2020), plausibly because the preconditions for generalization (Mutz Reference Mutz, Druckman and Green2021, 236) hold in my case: regarding treatments, the vignettes conceptually match real-world variation and are grounded in respondents’ lived experiences, as also verified by manipulation tests; while outcome measures are proximate across data types, and the treatments explain substantial variation in them – limiting the potential for confounding factors.
Beyond the specific case, these findings carry implications for a broader debate about what survey experiments can teach us. Some scholars are broadly optimistic: given their internal validity, survey experiments can at least illuminate theoretical micro-foundations (Mutz Reference Mutz2011; Pepinsky Reference Pepinsky, Box-Steffensmeier, Christenson and Sinclair-Chapman2023). Others argue they are hardly active treatments in the sense of Pearl (Reference Pearl2009) and provide little guidance for real-world questions or even policy (Samii Reference Samii, Box-Steffensmeier, Christenson and Sinclair-Chapman2024). In my case, I let respondents indicate how they would behave in a hypothetical, alternative reality (Brutger et al. Reference Brutger, Kertzer, Renshon, Tingley and Weiss2023), rather than actually manipulating their effective perceptions of the real world. The latter would not be advisable in principle (given ethical problems, see Samii Reference Samii, Box-Steffensmeier, Christenson and Sinclair-Chapman2024), while the former could compound the doubts surrounding an “active” treatment from which we can extrapolate to the real world. But to what extent these hypotheticals can be informative ultimately depends on whether respondents can predict their potential outcomes, i.e., how they actually would react to different alternative realities – which is an empirical question. My results show that, on the one hand, quantifying actual behavioral responses – even from vignettes exhibiting “mundane realism” (Druckman Reference Druckman2022, Ch. 3) – comes with strong requirements, the key barrier being an intention–behavior gap that goes beyond design choice (Sheeran and Webb Reference Sheeran and Webb2016).Footnote 10 On the other hand, hypothetical intentions replicate actual intentions well, and I observe broad sign-generalization for both intent and behavior. In that sense, hypothetical vignette experimental scenarios can function akin to active treatments and can allow respondents to indicate their actual potential outcomes. Hence, they can also provide evidence in the spirit of the credibility revolution, both at lower implementation costs, and in instances where exogenous variation may not be available in the real world – at least as long as the scenarios are plausibly meaningful even in an artificial (survey) environment (Levitt and List Reference Levitt and List2007), while the design explicitly verifies that respondents relate to the hypotheticals (Mutz and Kim Reference Mutz, Kim and Erin2020).
Future research could extend this work in multiple directions, also addressing limitations of my case. Findley, Kikuta, and Denly (Reference Findley, Kikuta and Denly2021) discuss generalizability across units, mechanisms, settings, and time – dimensions I can partly consider (e.g., the time dimension), but partly lack sufficient variation. Future research could also align concepts even more closely than was possible here. While my survey and real-world treatments have a direct relation, especially for the damage dimension, assessing effect-generalization comes with several conceptual limitations (Diaz, Grady, and Kuklinski Reference Diaz, Grady, Kuklinski, Curini and Franzese2020): pre-treated respondents, even if balanced across survey-experimental conditions, can bias baseline comparisons; treatment perception heterogeneity may not be comparable across data types; and factors not included in the experiment might moderate the core relationship, especially if they co-occur with the treatment in the real world (e.g., community assistance after exposure, or pandemic-related movement restrictions (Rudolph, Koubi, and Freihardt Reference Rudolph, Koubi and Freihardt2024)). Moreover, future research could investigate effect size comparisons based on objective rather than self-reported affectedness indicators, given that perceptions exhibit biases (Olken Reference Olken2009).Footnote 11
My results contribute to a small but growing literature of within-study comparisons on the external validity of survey-experimental treatments (e.g., Findley et al. Reference Findley, Laney, Nielson and Sharman2017; Hainmueller, Hangartner, and Yamamoto Reference Hainmueller, Hangartner and Yamamoto2015; Hoffmann, Kanitsar, and Seifert Reference Hoffmann, Kanitsar and Seifert2024; Petzold and Wolbring Reference Petzold and Wolbring2018), adding a rarely validated case from the Global South (for exceptions, see Auerbach and Thachil Reference Auerbach and Thachil2018; Whiting and Ma Reference Whiting and Ma2021). Together with related evidence that survey experiments generalize over differing contexts (Bassan-Nygate et al. Reference Bassan-Nygate, Renshon, Weeks and Weiss2025) or samples (Coppock, Leeper, and Mullinix Reference Coppock, Leeper and Mullinix2018), this is encouraging for their broader usability in testing social science theories (Blair, Coppock, and Humphreys Reference Blair, Coppock and Humphreys2023; Pepinsky Reference Pepinsky, Box-Steffensmeier, Christenson and Sinclair-Chapman2023).
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/XPS.2026.10035.
Data availability
The data and code required to replicate all analyses in this article are available at the Journal of Experimental Political Science Dataverse within the Harvard Dataverse Network, at https://doi.org/10.7910/DVN/CON75O.
Acknowledgements
I thank the editor and three excellent reviewers, as well as Macartan Humphreys for very helpful feedback on the manuscript. Participants at the 2025 Annual Meeting of the German Political Science Association’s Methods Section and at the 2026 PolMeth Europe Meeting provided helpful comments. The study design for survey data collection was jointly pre-registered with V. Koubi and J. Freihardt (ETH Zurich). I thank J. Freihardt and V. Koubi for joint questionnaire development and, particularly J. Freihardt, for leading the field implementation of the survey within a larger project. I thank our local enumerator team in Bangladesh for excellent fieldwork in data collection. I thank S. Zug for valuable discussions on the study context. K. Panther, E. Preiss, and S. Känner provided excellent research assistance.
Author contribution
The author confirms sole responsibility for the conception of the study, the presented results, and manuscript preparation.
Funding statement
The author discloses financial support by Swiss National Science Foundation Grant No. 185210 (Co-PI: LR; PI: V. Koubi, ETH Zurich).
Competing interests
The author reports no potential competing interests.
Ethics statement
The study was approved by the Ethics Commission of ETH Zurich (No. EK 2020-N-67). The author affirms this research adheres to APSA’s Principles and Guidance for Human Subjects Research. Respondents provided informed consent and were compensated for their time and to incentivize unbiased survey take-up (in-person survey, approx. 1 hour: 100 BDT; phone surveys, approx. 15 minutes: 30 BDT).
Open access
