Hostname: page-component-848d4c4894-x5gtn Total loading time: 0 Render date: 2024-05-06T02:06:03.467Z Has data issue: false hasContentIssue false

STRETCHING THE NET: MULTIDIMENSIONAL REGULARIZATION

Published online by Cambridge University Press:  05 August 2021

Jaume Vives-i-Bastida*
Affiliation:
Massachusetts Institute of Technology
*
Address correspondence to Jaume Vives-i-Bastida, MIT Department of Economics, 50 Memorial Dr, Cambridge, MA 02142, USA; e-mail: vives@mit.edu.

Abstract

This paper derives asymptotic risk (expected loss) results for shrinkage estimators with multidimensional regularization in high-dimensional settings. We introduce a class of multidimensional shrinkage estimators (MuSEs), which includes the elastic net, and show that—as the number of parameters to estimate grows—the empirical loss converges to the oracle-optimal risk. This result holds when the regularization parameters are estimated empirically via cross-validation or Stein’s unbiased risk estimate. To help guide applied researchers in their choice of estimator, we compare the empirical Bayes risk of the lasso, ridge, and elastic net in a spike and normal setting. Of the three estimators, we find that the elastic net performs best when the data are moderately sparse and the lasso performs best when the data are highly sparse. Our analysis suggests that applied researchers who are unsure about the level of sparsity in their data might benefit from using MuSEs such as the elastic net. We exploit these insights to propose a new estimator, the cubic net, and demonstrate through simulations that it outperforms the three other estimators for any sparsity level.

Type
MISCELLANEA
Copyright
© The Author(s), 2021. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

I thank Rachael Meager for invaluable guidance provided throughout the supervision of this paper at the London School of Economics, and Alberto Abadie, Maximilian Kasy, and Matthew Levy for giving insightful comments and feedback. I also thank David Card for starting me on this line of research. A co-editor of this journal and three anonymous referees provided very useful comments that contributed to improve this paper.

References

REFERENCES

Abadie, A. & Kasy, A. (2019) Choosing among regularized estimators in empirical economics: The risk of machine learning. The Review of Economic and Statistics 101(5), 743762.CrossRefGoogle Scholar
Chernozhukov, V., Hansen, C. & Liao, Y. (2017) A lava attack on the recovery of sums of dense and sparse signals. Annals of Statistics 45(1), 3976.CrossRefGoogle Scholar
Fan, J. & Li, R. (2001) Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association 96(456), 13481360.CrossRefGoogle Scholar
James, W. & Stein, C. (1961) Estimation with quadratic loss. In ed. Neyman, J., Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, pp. 361379. University of California.Google Scholar
Jia, J. & Yu, B. (2010) On model selection consistency of the elastic net when p ≫ n . Statistica Sinica 20, 595611.Google Scholar
Leeb, H. & Potscher, B. M. (2006) Performance limits for estimators of the risk or distribution of shrinkage-type estimators, and some general lower risk-bound results. Econometric Theory 22(1), 6997.CrossRefGoogle Scholar
Morris, C. N. (1983) Parametric empirical Bayes inference: Theory and applications. Journal of the American Statistical Association 78(381), 4755.CrossRefGoogle Scholar
Robbins, H. (1956) An empirical Bayes approach to statistics. In ed. Neyman, J., Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, pp. 157163. University of California Press.Google Scholar
Stein, C. M. (1956) Inadmissibility of the usual estimator for the mean of a multivariate distribution. In ed. Neyman, J., Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, pp. 197206. University of California Press.Google Scholar
van der Vaart, A. (1998) Asymptotic Statistics Cambridge University Press.CrossRefGoogle Scholar
Zou, H. (2006) The adaptive LASSO and its oracle properties. Journal of the American Statistical Association 101(476), 14181429.CrossRefGoogle Scholar
Zou, H. & Hastie, T. (2005) Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67, 301320.CrossRefGoogle Scholar