Skip to main content
×
×
Home

Estimation of Heterogeneous Treatment Effects from Randomized Experiments, with Application to the Optimal Planning of the Get-Out-the-Vote Campaign

  • Kosuke Imai (a1) and Aaron Strauss (a2)
Abstract

Although a growing number of political scientists are conducting randomized experiments, many of them only report the average treatment effects and do not systematically explore the variation in treatment effects across subpopulations. This is unfortunate from a scientific point of view because heterogeneous treatment effects can provide additional substantive insights. This current state of affairs is also problematic from a policy makers' perspective since such studies do not identify subgroups for which treatments are effective. In this paper, we propose a formal two-step framework that first identifies heterogeneous treatment effects from a randomized experiment and then uses this information to derive an optimal policy about which treatment should be given to whom. Our proposed method avoids the risk of false discoveries that are likely in post hoc subgroup analysis routinely conducted in the discipline. We discuss our methodology in the context of get-out-the-vote randomized field experiments and show how the proposed two-step framework can be applied in real-world settings.

    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Estimation of Heterogeneous Treatment Effects from Randomized Experiments, with Application to the Optimal Planning of the Get-Out-the-Vote Campaign
      Available formats
      ×
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Estimation of Heterogeneous Treatment Effects from Randomized Experiments, with Application to the Optimal Planning of the Get-Out-the-Vote Campaign
      Available formats
      ×
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Estimation of Heterogeneous Treatment Effects from Randomized Experiments, with Application to the Optimal Planning of the Get-Out-the-Vote Campaign
      Available formats
      ×
Copyright
Corresponding author
e-mail: kimai@princeton.edu (corresponding author)
Footnotes
Hide All

Authors' note: The first version of this paper was circulated in December 2008 under the title of “Planning the Optimal Get-out-the-vote Campaign.” We thank useful comments from seminar participants at Columbia University and the University of Wisconsin, Madison, as well as three anonymous reviewers and the editor. Supplementary materials for this article are available on the Political Analysis Web site.

Footnotes
References
Hide All
Arceneaux, Kevin, and Kolodny, Robin. 2009. Educating the least informed: Group endorsements in a grassroots campaign. American Journal of Political Science 53: 755–70.
Berger, James O. 1985. Statistical decision theory and Bayesian analysis. 2nd ed. New York: Springer.
Breiman, Leo, Friedman, Jerome, Stone, Charles J., and Olshen, R. A. 1984. Classification and regression trees. London: Chapman & Hall.
Chipman, Hugh A., George, Edward I., and McCulloch, Robert E. 2010. BART: Bayesian additive regression trees. Annals of Applied Statistics 4: 266–98.
Cole, Stephen R., and Stuart, Elizabeth A. 2010. Generalizing evidence from randomized clinical trials to target populations: The ACTG 320 trial. American Journal of Epidemiology 172: 107–15.
Cox, David R. 1958. Planning of experiments. New York: John Wiley & Sons.
Dale, Allison, and Strauss, Aaron. 2008. Mobilizing the mobiles: How text messaging can boost turnout. In Politicking online: The transformation of election campaign communications, ed. Panagopoulos, Costas, 152–62. New Brunswick, NJ: Rutgers University Press.
Dale, Allison, and Strauss, Aaron. 2009. Don't forget to vote: Text message reminders as a mobilization tool. American Journal of Political Science 53: 787804.
Dehejia, Rajeev. 2005. Program evaluation as a decision problem. Journal of Econometrics 125: 141–73.
Druckman, James N., Green, Donald P., Kuklinski, James H., and Lupia, Arthur. 2006. The growth and development of experimental research in political science. American Political Science Review 100: 627–35.
Eldersveld, Samuel J. 1956. Experimental propaganda techniques and voting behavior. American Political Science Review 50: 154–65.
Feller, Avi, and Holmes, Chris C. 2009. Beyond toplines: Heterogeneous treatment effects in randomized experiments. Technical report. Oxford, UK: University of Oxford.
Gerber, Alan S., and Green, Donald P. 2000. The effects of canvassing, telephone calls, and direct mail on voter turnout: A field experiment. American Political Science Review 94: 653–63.
Gerber, Alan S., Green, Donald P., and Larimer, Christopher W. 2008. Social pressure and voter turnout: Evidence from a large-scale field experiment. American Political Science Review 102: 3348.
Gerber, Alan S., Green, Donald P., and Shachar, Ron. 2003. Voting may be habit-forming: Evidence from a randomized field experiment. American Journal of Political Science 47: 540–50.
Gosnell, Harold F. 1927. Getting-out-the-vote: An experiment in the stimulation of voting. Chicago, IL: University of Chicago Press.
Green, Donald P., and Gerber, Alan S. 2008. Get out the vote: How to increase voter turnout. 2nd ed. Washington, DC: Brookings Institution.
Green, Donald P., and Kern, Holder L. 2010a. Generalizing experimental results. The Annual Meeting of the American Political Science Association, Washington, DC.
Green, Donald P., and Kern, Holder L. 2010b. Detecting heterogenous treatment effects in large-scale experiments using Bayesian additive regression trees. The Annual Summer Meeting of the Society of Political Methodology, University of Iowa.
Gunter, Lacey, Zhu, Ji, and Murphy, Susan. 2007. Variable selection for optimal decision making. Proceedings of the 11th Conference on Artificial Intelligence in Medicine AIME 2007, LNCS/LNAI 4594: 149–54.
Hartman, Erin, Grieve, Richard, and Sekhon, Jasjeet S. 2010. From SATE to PATT: The essential role of placebo test combining experimental and observational studies. The Annual Meeting of the American Political Science Association, Washington, DC.
Hartmann, George W. 1936. A field experiment on the comparative effectiveness of “emotional” and “rational” political leaflets in determining election results. Journal of Abnormal Psychology 31: 99114.
Hill, Jennifer L. Forthcoming. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics doi:10.1198/jcgs.2010.08162.
Horiuchi, Yusaku, Imai, Kosuke, and Taniguchi, Naoko. 2007. Designing and analyzing randomized experiments: Application to a Japanese election survey experiment. American Journal of Political Science 51: 669–87.
Kramer, G. H. 1966. A decision-theoretic analysis of a problem in political campaigning. In Mathematical applications in political science. Vol. II, ed. Bernd, J. L., 137–60. Dallas, TX: Southern Methodist University.
Lovett, Mitch, and Peress, Michael. 2010. Targeting political advertising on television. Technical report. Rochester, NY: University of Rochester.
Manski, Charles F. 2005. Social choice with partial knowledge of treatment response. Princeton, NJ: Princeton University Press.
Morton, Rebecca B., and Williams, Kenneth C. 2010. Experimental political science and the study of causality: From nature to the lab. New York: Cambridge University Press.
Nickerson, David W. 2008. Is voting contagious?: Evidence from two field experiments. American Political Science Review 102: 4957.
Nickerson, David W., and Arceneaux, Kevin. 2009. Who is mobilized to vote? A re-analysis of eleven randomized field experiments. American Journal of Political Science 53: 116.
Pocock, Stuart J., Assmann, Susan E., Enos, Laura E., and Kasten, Linda E. 2002. Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practice and problems. Statistics in Medicine 21: 2917–30.
Ripley, Brian D. 1996. Pattern recognition and neural networks. Cambridge: Cambridge University Press.
Rubin, Donald B. 1990. Comments on “On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9” by J. Splawa-Neyman translated from the Polish and edited by Dabrowska, D.M. and Speed, T.P. Statistical Science 5: 472–80.
Savage, L. J. 1951. The theory of statistical decision. Journal of the American Statistical Association 46: 5567.
Stoye, Jörg. 2009. Minimax regret treatment choice with finite samples. Journal of Econometrics 151: 7081.
Wald, Abraham. 1950. Statistical decision functions. New York: Wiley.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
MathJax
Type Description Title
PDF
Supplementary materials

Imai and Strauss supplementary material
Appendix

 PDF (67 KB)
67 KB

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed