Skip to main content

Estimating Heterogeneous Treatment Effects and the Effects of Heterogeneous Treatments with Ensemble Methods

  • Justin Grimmer (a1), Solomon Messing (a2) and Sean J. Westwood (a3)

Randomized experiments are increasingly used to study political phenomena because they can credibly estimate the average effect of a treatment on a population of interest. But political scientists are often interested in how effects vary across subpopulations—heterogeneous treatment effects—and how differences in the content of the treatment affects responses—the response to heterogeneous treatments. Several new methods have been introduced to estimate heterogeneous effects, but it is difficult to know if a method will perform well for a particular data set. Rather than using only one method, we show how an ensemble of methods—weighted averages of estimates from individual models increasingly used in machine learning—accurately measure heterogeneous effects. Building on a large literature on ensemble methods, we show how the weighting of methods can contribute to accurate estimation of heterogeneous treatment effects and demonstrate how pooling models lead to superior performance to individual methods across diverse problems. We apply the ensemble method to two experiments, illuminating how the ensemble method for heterogeneous treatment effects facilitates exploratory analysis of treatment effects.

Corresponding author
* Email:
Hide All

Authors’ note: Replication data available in Grimmer, Messing, and Westwood (2017).

Contributing Editor: Dustin Tingley

Hide All
Athey Susan, and Imbens Guido. 2015. Machine learning methods for estimating heterogeneous causal effects. Preprint, arXiv:1504.01132.
Berinsky Adam J., Huber Gregory A., and Lenz Gabriel S.. 2012. Evaluating online labor markets for experimental research:’s mechanical turk. Political Analysis 20:351368.
Breiman Leo. 2001. Random forests. Journal of Machine Learning 45(1):532.
Chatterjee Arindam, and Lahiri Soumendra Nath. 2011. Bootstrapping lasso estimators. Journal of the American Statistical Association 106(494):608625.
Chipman Hugh A., George Edward I., and McCulloch Robert E.. 2010. BART: Bayesian additive regression trees. Annals of Applied Statistics 41(1):266298.
Dietterich Thomas. 2000. Ensemble methods in machine learning. In Multiple Classifier Systems. MCS 2000 . Lecture Notes in Computer Science, vol. 1857. Heidelberg: Springer-Verlag.
Efron Bradley, and Tibshirani Robert J.. 1994. An introduction to the bootstrap . Boca Raton, FL: CRC Press.
Fong Christian, and Grimmer Justin. 2016. Discovery of treatments from text corpora. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin, Germany .
Gelman Andrew, Jakulin Aleks, Pittau Maria Grazia, and Su Yu-Sung. 2008. A weakly informative default prior distribution for logistic and other regression models. The Annals of Applied Statistics 2(4):13601383.
Gelman Andrew, Hill Jennifer, and Yajima Masanao. 2012. Why we (usually) don’t have to worry about multiple comparisons? Journal of Research on Educational Effectiveness 5(1):189211.
Gerber Alan S., and Green Donald P.. 2012. Field experiment: Design, analysis, and interpretation . New York: W.W. Norton & Company.
Green Donald P., and Kern Holger L.. 2012. Modeling heterogeneous treatment effects in survey experiments with Bayesian additive regression trees. Public Opinion Quarterly 76(3):491511.
Grimmer Justin. 2013. Representational style: What legislators say and why it matters . Cambridge: Cambridge University Press.
Grimmer Justin, Westwood Sean J., and Messing Solomon. 2014. The impression of influence: Legislator communication, representation, and democratic accountability . Princeton, NJ: Princeton University Press.
Grimmer Justin, Messing Solomon, and Westwood Sean J.. 2012. How words and money cultivate a personal vote: The effect of legislator credit claiming on constituent credit allocation. American Political Science Review 106(4):703719.
Grimmer Justin, Messing Solomon, and Westwood Sean. 2017. Replication data for estimating heterogeneous treatment effects and the effects of heterogeneous treatments with ensemble methods. doi:10.7910/DVN/BQMLQW.
Hainmueller Jens, and Hazlett Chad. 2013. Kernel regularized least squares: Reducing misspecification bias with a flexible and interpretable machine learning approach. Political Analysis 22(2):143168.
Hainmueller Jens, Hopkins Daniel, and Yamamoto Teppei. 2014. Causal inference in conjoint analysis: Understanding multi-dimensional choices via stated preference experiments. Political Analysis 22(1):130.
Hainmueller Jens, and Hopkins Daniel J.. 2015. The hidden American immigration consensus: A conjoint analysis of attitudes toward immigrants. American Journal of Political Science 59(3):529548.
Hartman Erin, Grieve Richard, Ramshai Roland, and Sekhon Jasjeet S.. 2012. From SATE to PATT: Combining experimental with observational studies. University of California, Berkeley Mimeo.
Hastie Trevor, Tibshirani Robert, and Friedman Jerome. 2001. The elements of statistical learning . Springer.
Hillard Dustin, Purpura Stephen, and Wilkerson John. 2008. Computer-assisted topic classification for mixed-methods social science research. Journal of Information Technology & Politics 4(4):3146.
Holland Paul. 1986. Statistics and causal inference. Journal of the American Statistical Association 81(396):945960.
Humphreys Macartan, de la Sierra Raul Sanchez, and van der Windt Peter. 2013. Fishing, commitment, and communication: A proposal for comprehensive nonbinding research registration. Political Analysis 21(1):120.
Imai Kosuke, and Strauss Aaron. 2011. Estimation of heterogeneous treatment effects from randomized experiments, with application to the optimal planning of the get-out-the-vote campaign. Political Analysis 19(1):119.
Imai Kosuke, and Ratkovic Marc. 2013. Estimating treatment effect heterogeneity in randomized program evaluation. The Annals of Applied Statistics 7(1):443470.
Kasperowicz Pete. 2013. GOP seeks planned parenthood study with hope to strip funding.
Keerthi S. S., Shevade S. K., Bhattacharyya C., and Murthy K. R. K.. 2001. Improvements to platt’s SMO algorithm for SVM classifier design. Neural Computation 13(3):637649.
King Gary, Tomz Michael, and Wittenberg Jason. 2000. Making the most of statistical analyses: Improving interpretation and presentation. American Journal of Political Science 44(2):347361.
Mayhew David. 1974. Congress: The electoral connection . New Haven, CT: Yale University Press.
Montgomery Jacob M., Hollenbach Florian M., and Ward Michael D.. 2012. Improving predictions using ensemble Bayesian model averaging. Political Analysis 20(3):271291.
Platt J. 1998. Fast training of support vector machines using sequential minimal optimization. In Advances in Kernel Methods - Support Vector Learning , ed. Schoelkopf B., Burges C., and Smola A.. Cambridge, MA: MIT Press, jplatt/smo.html.
Raftery Adrian E., Gneiting Tilmann, Balabdaoui Fadoua, and Polakowski Michael. 2005. Using Bayesian model averaging to calibrate forecast ensembles. Monthly Weather Review 133:11551174.
Ratkovic Marc, and Tingley Dustin. 2017. Sparse estimation and uncertainty with application to subgroup analysis. Political Analysis 25(1):140.
Samii Cyrus, Paler Laura, and Daly Sarah. 2017. Retrospective causal inference with machine learning ensembles: An application to anti-recidivism policies in colombia. Political Analysis 24(4):434456.
Skocpol Theda, and Williamson Vanessa. 2011. The tea party and the remaking of republican conservatism . Oxford: Oxford University Press.
van der Laan Mark, Polley Eric, and Hubbard Alan. 2007. Super learner. Statistical Applications in Genetics and Molecular Biology 6(1):121.
Van der Laan Mark J, and Rose Sherri. 2011. Targeted learning: Causal inference for observational and experimental data . New York: Springer Science & Business Media.
Wager Stefan, and Athey Susan. 2017. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, forthcoming.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Political Analysis
  • ISSN: 1047-1987
  • EISSN: 1476-4989
  • URL: /core/journals/political-analysis
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
Type Description Title
Supplementary materials

Grimmer et al supplementary material
Online Appendix

 Unknown (218 KB)
218 KB


Altmetric attention score

Full text views

Total number of HTML views: 63
Total number of PDF views: 502 *
Loading metrics...

Abstract views

Total abstract views: 1134 *
Loading metrics...

* Views captured on Cambridge Core between 4th September 2017 - 25th February 2018. This data will be updated every 24 hours.