Skip to main content Accessibility help


  • Access


      • Send article to Kindle

        To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Think Ahead: Cost Discounting and External Validity in Foreign Policy Survey Experiments
        Available formats

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Think Ahead: Cost Discounting and External Validity in Foreign Policy Survey Experiments
        Available formats

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Think Ahead: Cost Discounting and External Validity in Foreign Policy Survey Experiments
        Available formats
Export citation


This paper considers the implications of construal level theory in the context of survey experiments probing foreign policy opinion formation. Psychology research demonstrates that people discount the long-term consequences of decisions, thinking about distal or hypothetical events more abstractly than immediate scenarios. I argue that this tendency introduces a bias into survey experiments on foreign policy opinion. Respondents reasoning about an impending military engagement are likelier to consider its costs than are those reasoning in the abstract hypothetical environment. I provide evidence of this bias by replicating a common audience costs experimental design and introducing a prompt to consider casualties. I find that priming respondents to articulate their expectations about casualties in a foreign intervention reduces support and dampens the experimental effect, thereby cutting the estimated absolute audience cost substantially. This result suggests a gap between how survey respondents approach hypothetical and real situations of military intervention.


I would like to thank Nicholas Weller for his tireless help, extensive brainstorming, and numerous read-throughs in developing this project. I also appreciate the helpful comments from Patrick James, Youssef Chouhoud, Tom Jamieson, and all the participants in the University of Southern California's Center for International Studies’ working paper series. I also greatly appreciate the grant provided by USC-CIS to carry out the experiment. The data, code, and any additional materials required to replicate all analyses in this article are available at the Journal of Experimental Political Science Dataverse within the Harvard Dataverse Network, at: doi: 10.7910/DVN/5KM8VO. I had no conflicts of interest in the creation of this article.

Experimental tests of opinion in foreign crises have repeatedly shown that survey respondents punish inconsistency in presidential actions (Chaudoin 2014; Levendusky and Horowitz 2012; Potter and Baum 2014; Tomz 2007). Some have directly manipulated the cost of the hypothetical conflict so that all subjects have similar expectations (Kertzer and Brutger 2016; Trager and Vavreck 2011), but most audience cost experiments have not explored respondents’ default expectations about costs.

However, psychology research on construal level theory (CLT) demonstrates that people tend to discount the long-term consequences of decisions, and think about distal or hypothetical events more abstractly than they do immediate scenarios. In this paper, I argue that this tendency introduces a bias into survey experiments on foreign policy opinion about military intervention. This is because survey respondents reasoning about an impending military engagement are likelier to consider the costs of such an action than are those reasoning in the hypothetical environment. To provide evidence of this bias, I replicate the common “audience costs” experiment and introduce a prompt to consider casualties. I find that priming respondents to articulate their expectations about casualties in a foreign intervention cuts the estimated audience cost by more than half, reduces support for intervention in general, and substantially decreases disapproval of the hypothetical empty threat.


Audience Cost Theory (ACT), founded on the theoretical work of Schelling (1960) and Fearon (1994), holds that democratic audiences care about national reputation, and so will punish leaders who back down from military threats. The theory is difficult to test, so has inspired innovative experimental studies. Tomz (2007) first provided experimental support for the theory's assumption that audiences punish empty threats. Further experiments on audience costs have isolated other key variables in the causal logic of ACT: urgency of the crisis (Tomz 2007); ambiguity of the threat and party reputation (Trager and Vavreck 2011); reasons for backing down (Levendusky and Horowitz 2012); and other inconsistency costs (Levy et al. 2015). There have been other similarly designed ACT survey experiments.

An issue that experimenters have not adequately addressed in these designs is that people think differently about hypothetical and distant situations than they do about those present and immediate. In psychology, this is often discussed in the context of CLT (Bar-Anan et al. 2006; Liberman and Trope 1998; Trope 2012; Trope and Liberman 2010). In more distal scenarios, people engage in “high-level construal,” considering the situation abstractly, overlooking detail and focusing on general themes and desirability of outcomes (Trope 2012). By contrast, in more immediate scenarios, people engage in “low-level construal,” with increased consideration of the specific details and feasibility of outcomes (Trope and Liberman 2000). The common example is that contemplating a vacation six months away may involve thinking about “relaxing” and “having fun,” whereas doing so a week beforehand may involve thinking about making dinner reservations and the stress of driving a rented car on foreign roads. This disparity is known to cause poor long-term planning and inaccurate predictions about one's own behavior and abilities in the more distant future (Liberman and Trope 1998).

Costs are associated with low-level construal. This is part of why people fall victim to what is known as “delay discounting” (Frederick et al. 2002; Green and Myerson 2004; Murphy et al. 2001; Odum 2011), the tendency to discount (or neglect entirely) consideration of the costs and rewards in the distant future or in hypothetical situations, giving them less weight than costs and rewards closer at hand. Along these lines, we know that people procrastinate thinking about costs (Frederick et al. 2002), do not predict their behavior well (DellaVigna and Malmendier 2006), and generally estimate value poorly (Gabaix et al. 2005).

This understanding of human psychology is relevant in the foreign policy survey experimental context because the hypothetical scenarios shown to respondents are distant, but the scenarios they are meant to reflect in the analogous political environment are often quite immediate. This is especially true of ACT experiments, which are meant to teach us something about how people reason about impending military engagement in foreign civil wars, a costly endeavor. Prior work tells us that pecuniary costs affect support for presidential belligerence (Flores-Macías and Kreps 2015; Geys 2010; Kriner et al. 2015). Moreover, casualty levels—“the most salient cost of war” (Gartner 2008, 105)—have repeatedly been found to influence popular support for foreign intervention (Boettcher and Cobb 2006; Eichenberg 2005; Gelpi et al. 2009; Karol and Miguel 2007; Kriner 2006; Walsh 2015). With this in mind, the vacation example's parallel in foreign policy opinion research is the possibility that the hypothetical scenario an experimental respondent reads might induce high-level, desirability concerns—like national reputation or leadership competence and consistency—while failing to account for low-level, feasibility concerns—higher taxes, American casualties, and other wartime stressors. Past research shows that an actual, impending intervention in a foreign war would induce these low-level concerns.

Some researchers have explored the effects of cost on opinion in ACT experiments. They directly manipulate the cost of an action within the vignettes themselves, inserting a definite quantity within the mind of the respondent and measuring the effect of that number (Flores-Macías and Kreps 2015; Kriner et al. 2015; Kriner and Shen 2012; Walsh 2015). However, in many of these experiments, costs are neither purposefully manipulated nor discussed at all, and the context of the survey vignette remains one of low information and high abstraction. As such, we would expect respondents to engage in high-level construal and be less likely to consider relevant costs (O'Donoghue and Rabin 1999).

Hence, there are good empirical reasons to expect a construal level bias in the context of the foreign policy survey experiment because it relies on low information vignettes. The scholarly work outlined above implies that people assessing real military intervention hold casualties in their minds as a relevant consideration to a greater extent than to those assessing a hypothetical scenario, which does not encourage such low-level construal. As such, testing for the presence and effect of this sort of bias in experimental estimates of audience costs is an important contribution. Moreover, knowing whether respondents primed to consider costs have different levels of disapproval for belligerence (Kertzer and Brutger 2016) or policy inconsistency (Chaudoin 2014) may reveal important differences between respondents’ approaches to real and hypothetical situations of military intervention.


I designed an experiment to establish whether respondents in survey experimental tests of foreign policy opinion discount the costs associated with hypothetical acts of intervention. The basic design was to induce low-level construal by introducing an open-ended question to consider casualties before measuring approval, and then compare the size of the replicated effect within that group to that of the control, which received no such prime prior to indicating approval.1 Rather than planting a specific casualty level in the vignette as some prior work has, I allowed the respondents to maintain whatever expectations they already held about foreign intervention. This better reflects what might happen in a poll in which survey respondents are asked their opinion on an impending act of intervention, such as polls of American opinion about intervention in Syria, in 2013 (CNN 2013; The Economist 2013).

My treatment was a single question asking how many casualties they expected in the hypothetical scenario. This prime made them consider casualties for a moment before answering the approval question. According to CLT, respondents in the control group will construe at the high level but not the low level, perhaps considering national reputation or leader's competence, but not the costs of a real intervention, such as lost lives. However, treated respondents are more likely consider them, as prior work shows they would in the “real world.”

If experimental respondents already consider the real costs of intervention in forming their responses, this inducement of low-level construal of the scenario should change nothing. However, if they skip cost thinking, we should expect to see some mitigation of audience costs. Following this logic, the key hypotheses are as follows:

Cost hypothesis. Respondents primed to consider American casualties before indicating approval will

  1. 1. indicate greater disapproval of intervention,

  2. 2. indicate less disapproval of backing down from threats,

  3. 3. generate a lower estimated audience cost.

The experimental test was straightforward. In October 2015, I recruited 1,512 Amazon Mechanical Turk users for an experiment modeled after Tomz (2007).2 All respondents were shown the same introductory statement and one vignette from Tomz, in which a dictator invades its neighbor to get more power and resources, it has a strong military, and its victory hurts the safety and economy of the United States.3 This group garnered the highest disapproval in his study and should represent the most difficult test for my hypotheses.

The respondents were assigned randomly to one of six groups in a 3×2 arrangement.4 They each encountered one of three presidential actions now standard among audience cost experiments: the president stays out of the conflict (the Stay Out condition); threatens intervention and backs down (empty threat); or threatens and intervenes (follow through). They also encountered an attention check. Respondents were asked for their approval/disapproval following the scenario:

Do you approve, disapprove, or neither approve nor disapprove of the way the U.S. President handled the situation?

  • Approve

  • Disapprove

  • Neither approve nor disapprove

I followed up with two questions to create a seven-point ordinal measure: strongly disapprove, disapprove, lean disapprove, neutral, lean approve, approve, and strongly approve.

The treatment group received the following question before answering the approval question:

How many U.S. troops would you expect were [to be] killed when [if] the president [had] decided to stop the invasion with military force?

  • Fewer than 50

  • 50–99

  • 100–499

  • 500–1,999

  • 2,000–4,999

  • 5,000–10,000

  • More than 10,000

This prompted them to consider the cost of intervention in terms of casualties, and report their expectations, just before they indicated their level of approval.

The control group answered the approval question first, providing the measure of the dependent variable before indicating their expectations about casualties. In the control group, the casualties question serves the role of a data collection instrument; its presence has no effect on the findings discussed below.


Figure 1 displays the differences in respondent estimates of casualties across vignettes. The stay out group expected higher casualties than the empty threat group, who expected more than the follow through group. Since casualties are not discussed or even alluded to in any of the three vignettes, one would expect pretreatment perceptions about casualties to be orthogonal to vignette assignment. However, vignette assignment clearly alters expectations, even without mentioning casualties.

Figure 1 Effect of Vignettes on Expected Casualties.

Dafoe et al. (2018) call this kind of variance in assumptions about undiscussed details in the vignettes an “information equivalence violation.” This kind of systematic variation in background assumptions can sometimes mediate experimental effects through atheoretical pathways. In addition to the construal level discrepancy identified in this study, the difference in respondent expectations about casualties identified here could also be responsible for some portion of the treatment effects found in previous experiments.5 Importantly, this possibility does not affect the validity of the comparisons made in this paper, which are based on assignment to a casualties prime, not to a specific scenario.

I analyzed differences in presidential approval across treatment groups in two ways: independent two-group t-tests using the seven-point measure of approval to obtain the initial comparison across the treatment groups;6 and the better suited ordered logit test, with standard demographic controls.7

Figure 2 shows that the casualties prime significantly increased average approval of the president in the empty threat condition, by about 0.4 points on our seven-point scale. The other two groups move in the expected direction, but are insignificant. Experimental studies of audience costs generally hinge upon changes within the empty threat group, so this result is pivotal.

Figure 2 Difference in Approval from Casualties Prime.

The ordered logit model provides an even clearer picture.8 The difference for the stay out group was negligible, but the empty threat and follow through groups saw a similarly oriented effect on approval of intervention, significant at the 0.05 level. Figure 3 shows the difference caused by the casualties prime on the predicted probability of assignment to each approval category. Respondents in the empty threat group, primed to consider casualties were: 5.9% less likely to strongly disapprove, 1.9% less likely to somewhat disapprove, 3.3% more likely to somewhat approve, and 2.7% more likely to strongly approve. Conversely, follow through respondents were 3.1% more likely to strongly disapprove, 1.9% more likely to somewhat disapprove, 2.1% less likely to somewhat approve, and 4.9% less likely to strongly approve. In short, primed subjects were more critical of intervention and less critical of inconsistency, supporting the first two components of the cost hypothesis.

Figure 3 Change in Predicted Assignment with Casualties Prime.

Most importantly, it sets up the hypothesis’ third component, because the muted disapproval of the empty threat altered the “absolute audience cost.” This was defined by Tomz as the “surge in disapproval” caused by not following through on the commitment to intervene, as compared to a commitment to stay out of the conflict altogether (Tomz 2007, 829).

Table 1 replicates Tomz's method of calculating absolute audience cost (Tomz 2007, 827), and demonstrates the difference in absolute cost caused by the casualties prime. With an absolute cost of 17 for the control group, my estimate approximates Tomz's estimate of 16. However, the casualties prime resulted in an 11-point drop in absolute audience cost, leaving it at just 6 for the treated group. Some discounting of casualties may be expected in these experiments, but that priming casualty consideration should alter the standard result so much is surprising. Prior work indicates expectations about casualties inform opinion on military engagement, so this result does indeed imply a gap between how subjects respond to real and hypothetical acts of military intervention.

Table 1 The Effect of Casualties Priming on Absolute Audience Costs

95% confidence interval in parentheses.

*Discrepancy due to rounding error.


Psychology research associated with CLT suggests that people tend to give less consideration to low-level concerns in abstract scenarios than they do in those closer at hand. Costs are subject to low-level construal and consideration of the feasibility of outcomes. However, abstract situations encourage primarily high-level construal and prioritizing concerns of desirability. In the survey experimental context, this pattern would seem to suggest the existence of a small but specific external validity problem: that subjects responding to the low information vignettes common in these designs would give relevant costs less consideration than they would if they were facing the analogous situation of impending American intervention in the real world.

Two findings reported herein affirm cause for this concern. First, I find an effect consistent with expectations derived from CLT and delay discounting. Respondents asked to estimate casualties support a hypothetical act of intervention to a significantly lesser degree than those who are not. Inducing them to think about casualties increases approval of backing down from commitments and decreases approval of following through on threats. This is despite the fact that they reported a relatively lower expected casualty level in the follow through scenario.

Second, the casualties prime substantially decreased estimated audience costs. This might imply that some prior experimental results on audience costs are less reflective of real-world opinion formation on foreign intervention than they are taken to be. Citizens assessing real military intervention consider casualties. Inducing them to do so in the hypothetical is shown here to strongly mute the size of the estimated audience cost the president faces.9

Two methodological recommendations are implied by these results. First, to improve external validity, experimentalists embedding their treatments in vignettes would benefit from considering possible gaps in construal level between their scenario and the analogous political environment, and then design features to encourage the right level of consideration. This may take the form of a similar question to the one employed in this design, a simple suggestion in the main prompt to consider the costs and benefits, or explicit information about low-level issues embedded in the vignette.

Second, respondents have different assumptions about casualties across treatment groups, even though they are mentioned nowhere in the initial scenario. The presence of an information equivalence violation in this replication study may also suggest an internal validity threat in the basic hypothetical design. Without some evidence that the effect is carried through a theoretically outlined pathway, it is difficult to know how exactly to interpret it. Whenever possible, experimentalists should make sure to incorporate tests of causal mechanisms in their work.

Several useful avenues for future work are implied in these findings. A project aimed at fully understanding these findings might identify whether and to what extent a similar prime to consider casualties would affect opinion on a real international military engagement. Another useful direction might be to explore untested respondent assumptions about the benefits of intervention, and test whether those are construed similarly to costs. Further work might also explore the extent to which information equivalence violations are present in or responsible for common experimental findings. Overall, there is much room to expand our understanding of respondent behavior in the experimental context within the foreign policy research agenda.


To view supplementary material for this article, please visit

1 As a comparative test to show that another prime does not cause this effect, and that casualties have particular salience in this environment, I also implemented a second test priming the respondents to consider the effect on America's international reputation, a high-level consideration. This prime had a minimal effect. See Appendix A.

2 The sample was limited to Mechanical Turk users in the United States.

3 Full survey instruments are in Appendix D.

4 The N for each group is in Appendix B.

5 Estimating the extent to which this sort of violation may alter or dampen the standard audience cost finding would require both a different design and other statistical tests, and is beyond the scope of this study. However, this finding is nonetheless intriguing and worth reporting, and it warrants further investigation in future scholarship.

6 Approval is treated as continuous. Full table of t-test results in Appendix B.

7 The parallel regressions assumption was tested with likelihood ratio, score, Wald, Wolfe–Gould, and Brant tests; and was found not to be violated. Controls were for age, sex, party, education level, and race. Full results in Appendix B.

8 These results are robust to an ordered probit test as well. See Appendix B for all models.

9 Furthermore, there is no equivalent effect from priming consideration of national reputation. See Appendix A.


Bar-Anan, Yoav, Liberman, Nira, and Trope, Yaacov. 2006. “The Association Between Psychological Distance and Construal Level: Evidence from an Implicit Association Test.” Journal of Experimental Psychology: General 135 (4): 609–22. doi: http://doi.org10.1037/0096-3445.135.4.609.
Boettcher, William A. and Cobb, Michael D.. 2006. “Echoes of Vietnam? Casualty Framing and Public Perceptions of Success and Failure in Iraq.” The Journal of Conflict Resolution 50 (6): 831–54.
Chaudoin, Stephen. 2014. “Audience Features and the Strategic Timing of Trade Disputes.” International Organization 68 (04): 877911. doi:
CNN. 2013. CNN | ORC Poll. (, accessed May 19, 2013.
Dafoe, Allan, Zhang, Baobao, and Caughey, Devin. 2018. “Informational Equivalence in Survey Experiments.” Political Analysis. doi:
DellaVigna, Stefano and Malmendier, Ulrike. 2006. “Paying Not to Go to the Gym.” The American Economic Review 96 (3): 694719.
Eichenberg, Richard C. 2005. “Victory Has Many Friends: U.S. Public Opinion and the Use of Military Force, 1981–2005.” International Security 30 (1): 140–77.
Fearon, James D. 1994. “Domestic Political Audiences and the Escalation of International Disputes.” American Political Science Review 88 (3): 577–92.
Flores-Macías, Gustavo A. and Kreps, Sarah E.. 2015. “Borrowing Support for War The Effect of War Finance on Public Attitudes toward Conflict.” Journal of Conflict Resolution 61 (5): 9971020.
Frederick, Shane, Loewenstein, George, and O'Donoghue, Ted. 2002. “Time Discounting and Time Preference: A Critical Review.” Journal of Economic Literature 40 (2): 351401.
Gabaix, Xavier, Li, Hongyi, and Laibson, David Isaac. 2005. Extreme Value Theory and the Effects of Competition on Profits. ( Accessed September 8, 2017.
Gartner, Scott Sigmund. 2008. “The Multiple Effects of Casualties on Public Support for War: An Experimental Approach.” The American Political Science Review 102 (1): 95106.
Gelpi, Christopher, Feaver, Peter, and Reifler, Jason Aaron. 2009. Paying the Human Costs of War: American Public Opinion and Casualties in Military Conflicts. Princeton: Princeton University Press.
Geys, Benny. 2010. “Wars, Presidents, and Popularity: The Political Cost(s) of War Re-Examined.” Public Opinion Quarterly 74 (2): 357–74. doi:
Green, Leonard and Myerson, Joel. 2004. “A Discounting Framework for Choice with Delayed and Probabilistic Rewards.” Psychological Bulletin 130 (5): 769–92. doi:
Huddleston, R. Joseph. 2018. “Replication Data for: Think Ahead: Cost Discounting and External Validity in Foreign Policy Survey Experiments.” Harvard Dataverse, V1. doi:, UNF:6:NeJwZwmlcAA29xwvccOXLA==
Karol, David and Miguel, Edward. 2007. “The Electoral Cost of War: Iraq Casualties and the 2004 U.S. Presidential Election.” The Journal of Politics 69 (3): 633–48. doi:
Kertzer, Joshua D. and Brutger, Ryan. 2016. “Decomposing Audience Costs: Bringing the Audience Back into Audience Cost Theory.” American Journal of Political Science 60 (1): 234–49. doi:
Kriner, Douglas L. 2006. “Examining Variance in Presidential Approval: The Case of FDR in World War II.” The Public Opinion Quarterly 70 (1): 2347.
Kriner, Douglas L. and Shen, Francis X.. 2012. “How Citizens Respond to Combat Casualties: The Differential Impact of Local Casualties on Support for the War in Afghanistan.” Public Opinion Quarterly 76 (4): 761–70. doi: 10.1093/poq/nfs048.
Kriner, Douglas, Lechase, Breanna, and Zielinski, Rosella Cappella. 2015. “Self-Interest, Partisanship, and the Conditional Influence of Taxation on Support for War in the USA.” Conflict Management and Peace Science 35 (1): 4364. doi:
Levendusky, Matthew S. and Horowitz, Michael C.. 2012. “When Backing Down Is the Right Decisions: Partisanship, New Information, and Audience Costs.” The Journal of Politics 74 (2): 323–38.
Levy, Jack S., McKoy, Michael K., Poast, Paul, and Wallace, Geoffrey P. R.. 2015. “Backing Out or Backing In? Commitment and Consistency in Audience Costs Theory.” American Journal of Political Science 59 (4): 9881001. doi:
Liberman, Nira and Trope, Yaacov. 1998. “The Role of Feasibility and Desirability Considerations in near and Distant Future Decisions: A Test of Temporal Construal Theory.” Journal of Personality and Social Psychology 75 (1): 518. doi:
Murphy, James G., Vuchinich, Rudy E., and Simpson, Cathy A.. 2001. “Delayed Reward and Cost Discounting.” The Psychological Record; Heidelberg 51 (4): 571–88.
Odum, Amy L. 2011. “Delay Discounting: I'm a k, You're a K.” Journal of the Experimental Analysis of Behavior 96 (3): 427–39. doi:
O'Donoghue, Ted and Rabin, Matthew. 1999. “Doing It Now or Later.” The American Economic Review 89 (1): 103–24.
Potter, Philip B. K. and Baum, Matthew A.. 2014. “Looking for Audience Costs in All the Wrong Places: Electoral Institutions, Media Access, and Democratic Constraint.” The Journal of Politics 76 (01): 167181. doi:
Schelling, Thomas C. 1960. The Strategy of Conflict. Cambridge, MA, USA: Harvard University Press.
The Economist. 2013. YouGov | Economist/YouGov Poll.( Accessed October 22, 2018.
Tomz, Michael. 2007. “Domestic Audience Costs in International Relations: An Experimental Approach.” International Organization 61 (4): 821–40.
Trager, Robert F. and Vavreck, Lynn. 2011. “The Political Costs of Crisis Bargaining: Presidential Rhetoric and the Role of Party.” American Journal of Political Science 55 (3): 526–45.
Trope, Yaacov. 2012. “Construal Level Theory.” In Handbook of Theories of Social Psychology, eds. Van Lange, Paul A. M., Kruglanski, Arie W., and Higgins, E. Tory. Los Angeles: SAGE, 118–34.
Trope, Yaacov and Liberman, Nira. 2000. “Temporal Construal and Time-Dependent Changes in Preference.” Journal of Personality and Social Psychology 79 (6): 876–89. doi:
Trope, Yaacov and Liberman, Nira. 2010. “Construal-Level Theory of Psychological Distance.” Psychological Review 117 (2): 440–63. doi:
Walsh, James Igoe. 2015. “Precision Weapons, Civilian Casualties, and Support for the Use of Force.” Political Psychology 36 (5): 507–23. doi: