Hostname: page-component-5db58dd55d-qmkzp Total loading time: 0 Render date: 2026-06-02T04:49:03.572Z Has data issue: false hasContentIssue false
  • English
  • Français

Estimation of Heterogeneous Treatment Effects from Randomized Experiments, with Application to the Optimal Planning of the Get-Out-the-Vote Campaign

Part of: PA editors' choice articles

Published online by Cambridge University Press:  04 January 2017

Kosuke Imai  and
Aaron Strauss
Kosuke Imai*
Affiliation:
Department of Politics, Princeton University, Princeton, NJ 08544
Aaron Strauss
Affiliation:
The Mellman Group, 1023 31st St NW, 5th Floor, Washington, DC 20007. e-mail: aaronbs@gmail.com
*
e-mail: kimai@princeton.edu (corresponding author)
Rights & Permissions [Opens in a new window]

Abstract

Core share and HTML view are not available for this content. However, as you have access to this content, a full PDF is available via the 'Save PDF' action button.

Although a growing number of political scientists are conducting randomized experiments, many of them only report the average treatment effects and do not systematically explore the variation in treatment effects across subpopulations. This is unfortunate from a scientific point of view because heterogeneous treatment effects can provide additional substantive insights. This current state of affairs is also problematic from a policy makers' perspective since such studies do not identify subgroups for which treatments are effective. In this paper, we propose a formal two-step framework that first identifies heterogeneous treatment effects from a randomized experiment and then uses this information to derive an optimal policy about which treatment should be given to whom. Our proposed method avoids the risk of false discoveries that are likely in post hoc subgroup analysis routinely conducted in the discipline. We discuss our methodology in the context of get-out-the-vote randomized field experiments and show how the proposed two-step framework can be applied in real-world settings.

Information

Type
Research Article
Information
Political Analysis , Volume 19 , Issue 1 , Winter 2011 , pp. 1 - 19
Copyright
Copyright © The Author 2010. Published by Oxford University Press on behalf of the Society for Political Methodology 

Footnotes

Authors' note: The first version of this paper was circulated in December 2008 under the title of “Planning the Optimal Get-out-the-vote Campaign.” We thank useful comments from seminar participants at Columbia University and the University of Wisconsin, Madison, as well as three anonymous reviewers and the editor. Supplementary materials for this article are available on the Political Analysis Web site.

References

Arceneaux, Kevin, and Kolodny, Robin. 2009. Educating the least informed: Group endorsements in a grassroots campaign. American Journal of Political Science 53: 755–70.CrossRefGoogle Scholar
Berger, James O. 1985. Statistical decision theory and Bayesian analysis. 2nd ed. New York: Springer.Google Scholar
Breiman, Leo, Friedman, Jerome, Stone, Charles J., and Olshen, R. A. 1984. Classification and regression trees. London: Chapman & Hall.Google Scholar
Chipman, Hugh A., George, Edward I., and McCulloch, Robert E. 2010. BART: Bayesian additive regression trees. Annals of Applied Statistics 4: 266–98.Google Scholar
Cole, Stephen R., and Stuart, Elizabeth A. 2010. Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial. American Journal of Epidemiology 172: 107–15.Google Scholar
Cox, David R. 1958. Planning of experiments. New York: John Wiley & Sons.Google Scholar
Dale, Allison, and Strauss, Aaron. 2008. Mobilizing the mobiles: How text messaging can boost turnout. In Politicking online: The transformation of election campaign communications, ed. Panagopoulos, Costas, 152–62. New Brunswick, NJ: Rutgers University Press.Google Scholar
Dale, Allison, and Strauss, Aaron. 2009. Don't forget to vote: Text message reminders as a mobilization tool. American Journal of Political Science 53: 787–804.Google Scholar
Dehejia, Rajeev. 2005. Program evaluation as a decision problem. Journal of Econometrics 125: 141–73.Google Scholar
Druckman, James N., Green, Donald P., Kuklinski, James H., and Lupia, Arthur. 2006. The growth and development of experimental research in political science. American Political Science Review 100: 627–35.Google Scholar
Eldersveld, Samuel J. 1956. Experimental propaganda techniques and voting behavior. American Political Science Review 50: 154–65.Google Scholar
Feller, Avi, and Holmes, Chris C. 2009. Beyond toplines: Heterogeneous treatment effects in randomized experiments. Technical report. Oxford, UK: University of Oxford.Google Scholar
Gerber, Alan S., and Green, Donald P. 2000. The effects of canvassing, telephone calls, and direct mail on voter turnout: A field experiment. American Political Science Review 94: 653–63.Google Scholar
Gerber, Alan S., Green, Donald P., and Larimer, Christopher W. 2008. Social pressure and voter turnout: Evidence from a large-scale field experiment. American Political Science Review 102: 33–48.Google Scholar
Gerber, Alan S., Green, Donald P., and Shachar, Ron. 2003. Voting may be habit-forming: Evidence from a randomized field experiment. American Journal of Political Science 47: 540–50.Google Scholar
Gosnell, Harold F. 1927. Getting-out-the-vote: An experiment in the stimulation of voting. Chicago, IL: University of Chicago Press.Google Scholar
Green, Donald P., and Gerber, Alan S. 2008. Get out the vote: How to increase voter turnout. 2nd ed. Washington, DC: Brookings Institution.Google Scholar
Green, Donald P., and Kern, Holder L. 2010a. Generalizing experimental results. The Annual Meeting of the American Political Science Association, Washington, DC.Google Scholar
Green, Donald P., and Kern, Holder L. 2010b. Detecting heterogenous treatment effects in large-scale experiments using Bayesian additive regression trees. The Annual Summer Meeting of the Society of Political Methodology, University of Iowa.Google Scholar
Gunter, Lacey, Zhu, Ji, and Murphy, Susan. 2007. Variable selection for optimal decision making. Proceedings of the 11th Conference on Artificial Intelligence in Medicine AIME 2007, LNCS/LNAI 4594: 149–54.Google Scholar
Hartman, Erin, Grieve, Richard, and Sekhon, Jasjeet S. 2010. From SATE to PATT: The essential role of placebo test combining experimental and observational studies. The Annual Meeting of the American Political Science Association, Washington, DC.Google Scholar
Hartmann, George W. 1936. A field experiment on the comparative effectiveness of “emotional” and “rational” political leaflets in determining election results. Journal of Abnormal Psychology 31: 99–114.Google Scholar
Hill, Jennifer L. Forthcoming. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics doi:10.1198/jcgs.2010.08162.Google Scholar
Horiuchi, Yusaku, Imai, Kosuke, and Taniguchi, Naoko. 2007. Designing and analyzing randomized experiments: Application to a Japanese election survey experiment. American Journal of Political Science 51: 669–87.Google Scholar
Kramer, G. H. 1966. A decision-theoretic analysis of a problem in political campaigning. In Mathematical applications in political science. Vol. II, ed. Bernd, J. L., 137–60. Dallas, TX: Southern Methodist University.Google Scholar
Lovett, Mitch, and Peress, Michael. 2010. Targeting political advertising on television. Technical report. Rochester, NY: University of Rochester.Google Scholar
Manski, Charles F. 2005. Social choice with partial knowledge of treatment response. Princeton, NJ: Princeton University Press.Google Scholar
Morton, Rebecca B., and Williams, Kenneth C. 2010. Experimental political science and the study of causality: From nature to the lab. New York: Cambridge University Press.Google Scholar
Nickerson, David W. 2008. Is voting contagious?: Evidence from two field experiments. American Political Science Review 102: 49–57.Google Scholar
Nickerson, David W., and Arceneaux, Kevin. 2009. Who is mobilized to vote? A re-analysis of eleven randomized field experiments. American Journal of Political Science 53: 1–16.Google Scholar
Pocock, Stuart J., Assmann, Susan E., Enos, Laura E., and Kasten, Linda E. 2002. Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practice and problems. Statistics in Medicine 21: 2917–30.Google Scholar
Ripley, Brian D. 1996. Pattern recognition and neural networks. Cambridge: Cambridge University Press.Google Scholar
Rubin, Donald B. 1990. Comments on “On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9” by J. Splawa-Neyman translated from the Polish and edited by Dabrowska, D.M. and Speed, T.P. Statistical Science 5: 472–80.CrossRefGoogle Scholar
Savage, L. J. 1951. The theory of statistical decision. Journal of the American Statistical Association 46: 55–67.Google Scholar
Stoye, Jörg. 2009. Minimax regret treatment choice with finite samples. Journal of Econometrics 151: 70–81.Google Scholar
Wald, Abraham. 1950. Statistical decision functions. New York: Wiley.Google Scholar
Supplementary material: PDF

Imai and Strauss supplementary material

Appendix

Download Imai and Strauss supplementary material(PDF)
PDF 67.3 KB