Hostname: page-component-848d4c4894-xm8r8 Total loading time: 0 Render date: 2024-06-15T13:21:22.767Z Has data issue: false hasContentIssue false

BARP: Improving Mister P Using Bayesian Additive Regression Trees — CORRIGENDUM

Published online by Cambridge University Press:  14 December 2022

MAX GOPLERUD
Affiliation:
University of Pittsburgh, United States
JAMES BISBEE
Affiliation:
Vanderbilt University, United States
Rights & Permissions [Opens in a new window]

Abstract

Type
Corrigendum
Copyright
© The Author(s), 2022. Published by Cambridge University Press on behalf of the American Political Science Association

Bisbee regrets the coding error in the above article. As identified by Goplerud (Reference GoplerudForthcoming) [Appendix E.1], Bisbee (Reference Bisbee2019a)’s replication code failed to sort the data prior to calculating predictions from the MRP model, leading to the injection of noise into MRP estimates while not affecting BARP estimates. This lead to exaggerated performance improvements when comparing traditional MRP with BARP. The error affects Figures 1, 2, and 3 in the original publication as well as figures in the supplementary materials. Bisbee has corrected figures for the main text which are reproduced below (similar figures appear in Goplerud Reference GoplerudForthcoming), and he has updated the associated supporting materials for Bisbee (Reference Bisbee2019a) to reflect this correction.

Figure 1. Predictive Accuracy

Note: Predictive accuracy of BARP (y-axes) versus MRP (x-axes) across 89 surveys as measured by mean absolute error (left panel) and interstate correlation (right panel).

Figure 2. Sensitivity to Misspecification

Note: Difference-in-means estimates (points) and confidence intervals (lines) indicating how much better MRP (x-axes) and BARP (y-axes) perform when the two state-level covariates are included. Negative values on the left-hand plot reflect smaller absolute errors in the full specification, whereas positive values on the right-hand plot reflect larger interstate correlations in the full specification.

Figure 3. Method Sensitivity to State Sample Size

Note: Coefficients (points) for each survey measuring the relationship between mean absolute error and the number of observations in the state for BARP (y-axis) and MRP (x-axis). Negative values indicate that more observations in a state improve improve mean absolute error by the units on the x and y-axes. Two standard errors indicated by horizontal and vertical lines. Values closer to zero (dashed lines) reflect greater insulation from data sparsity.

The corrected version of Figure 1 and its caption are provided below. The corrected results demonstrate that the difference in performance between the two methods is much more of a toss-up, whether evaluated using mean absolute error (MAE, left panel) or interstate correlation (right panel). Goplerud (Reference GoplerudForthcoming) summarizes the difference quantitatively by averaging across the surveys and reports a small improvement of BARP over (traditional) MRP of around 4.5% with a sample size of 1,500. When considering mean absolute error, it notes that this decreases to around 1% for larger sample sizes.

The corrected version of Figure 2 and its caption are provided below. It also shows a much more similar performance between the two methods across the range of surveys considered.

The corrected version of Figure 3 and its caption are provided below. It shows results that are consistent with the claim in Bisbee (Reference Bisbee2019a), i.e. that BARP is less sensitive to smaller sample sizes. Appendix E.2 of Goplerud (Reference GoplerudForthcoming) provides a different test of this claim and finds limited differences between the methods.

Conclusion

After the discovery the original error, the two authors of this corrigendum spoke, agreed on the source and nature of the error, and then jointly wrote this correction. Bisbee has re-examined the associated software for implementing BART for MRP and confirmed that the error did not affect recent research that has relied on this software.Footnote 1 Updated replication materials are available at Bisbee (Reference Bisbee2019b).

References

REFERENCES

Bisbee, James. 2019a. “BARP: Improving Mister P Using Bayesian Additive Regression Trees.” American Political Science Review 113(4): 10601065.CrossRefGoogle Scholar
Bisbee, James. 2019b. “Replication Data for: BARP: Improving Mister P Using Bayesian Additive Regression Trees, V2.” Harvard Dataverse. Dataset. https://doi.org/10.7910/DVN/LMW871.CrossRefGoogle Scholar
Goplerud, Max. Forthcoming. “Re-Evaluating Machine Learning for MRP Given the Comparable Performance of (Deep) Hierarchical Models.” American Political Science Review.Google Scholar
Figure 0

Figure 1. Predictive AccuracyNote: Predictive accuracy of BARP (y-axes) versus MRP (x-axes) across 89 surveys as measured by mean absolute error (left panel) and interstate correlation (right panel).

Figure 1

Figure 2. Sensitivity to MisspecificationNote: Difference-in-means estimates (points) and confidence intervals (lines) indicating how much better MRP (x-axes) and BARP (y-axes) perform when the two state-level covariates are included. Negative values on the left-hand plot reflect smaller absolute errors in the full specification, whereas positive values on the right-hand plot reflect larger interstate correlations in the full specification.

Figure 2

Figure 3. Method Sensitivity to State Sample SizeNote: Coefficients (points) for each survey measuring the relationship between mean absolute error and the number of observations in the state for BARP (y-axis) and MRP (x-axis). Negative values indicate that more observations in a state improve improve mean absolute error by the units on the x and y-axes. Two standard errors indicated by horizontal and vertical lines. Values closer to zero (dashed lines) reflect greater insulation from data sparsity.

Submit a response

Comments

No Comments have been published for this article.