Skip to main content
×
×
Home

ADAPTIVE BAYESIAN ESTIMATION OF CONDITIONAL DENSITIES

  • Andriy Norets (a1) and Debdeep Pati (a2)
Abstract

We consider a nonparametric Bayesian model for conditional densities. The model is a finite mixture of normal distributions with covariate dependent multinomial logit mixing probabilities. A prior for the number of mixture components is specified on positive integers. The marginal distribution of covariates is not modeled. We study asymptotic frequentist behavior of the posterior in this model. Specifically, we show that when the true conditional density has a certain smoothness level, then the posterior contraction rate around the truth is equal up to a log factor to the frequentist minimax rate of estimation. An extension to the case when the covariate space is unbounded is also established. As our result holds without a priori knowledge of the smoothness level of the true density, the established posterior contraction rates are adaptive. Moreover, we show that the rate is not affected by inclusion of irrelevant covariates in the model. In Monte Carlo simulations, a version of the model compares favorably to a cross-validated kernel conditional density estimator.

Copyright
Corresponding author
*Address correspondence to Andriy Norets, Associate Professor, Department of Economics, Brown University, Providence, RI 02912; e-mail: andriy_norets@brown.edu.
Footnotes
Hide All

We thank the editor, the co-editor, and referees for helpful comments. Dr. Pati acknowledges support for this project from the Office of Naval Research (ONR BAA 14-0001) and NSF DMS-1613156.

Footnotes
References
Hide All
Barron, A., Schervish, M.J., & Wasserman, L. (1999) The consistency of posterior distributions in nonparametric problems. The Annals of Statistics 27(2), 536561.
Bhattacharya, A., Pati, D., & Dunson, D. (2014) Anisotropic function estimation using multi-bandwidth Gaussian processes. The Annals of Statistics 42(1), 352381.
Chung, Y. & Dunson, D.B. (2009) Nonparametric Bayes conditional distribution modeling with variable selection. Journal of the American Statistical Association 104(488), 16461660.
De Iorio, M., Muller, P., Rosner, G.L., & MacEachern, S.N. (2004) An ANOVA model for dependent random measures. Journal of the American Statistical Association 99(465), 205215.
De Jonge, R. & van Zanten, J.H. (2010) Adaptive nonparametric Bayesian inference using location-scale mixture priors. The Annals of Statistics 38(6), 33003320.
Dunson, D.B. & Park, J.H. (2008) Kernel stick-breaking processes. Biometrika 95(2), 307323.
Dunson, D.B., Pillai, N., & Park, J.H. (2007) Bayesian density regression. Journal of the Royal Statistical Society Series B (Statistical Methodology) 69(2), 163183.
Efromovich, S. (2007) Conditional density estimation in a regression setting. The Annals of Statistics 35(6), 25042535.
Geweke, J. & Keane, M. (2007) Smoothly mixing regressions. Journal of Econometrics 138, 252290.
Ghosal, S., Ghosh, J.K., & Ramamoorthi, R.V. (1999) Posterior consistency of Dirichlet mixtures in density estimation. The Annals of Statistics 27(1), 143158.
Ghosal, S., Ghosh, J.K., & van der Vaart, A.W. (2000) Convergence rates of posterior distributions. The Annals of Statistics 28(2), 500531.
Ghosal, S. & van der Vaart, A.W. (2001) Entropies and rates of convergence for maximum likelihood and Bayes estimation for mixtures of normal densities. The Annals of Statistics 29(5), 12331263.
Ghosal, S. & van der Vaart, A.W. (2007) Posterior convergence rates of Dirichlet mixtures at smooth densities. The Annals of Statistics 35(2), 697723.
Griffin, J.E. & Steel, M.F.J. (2006) Order-based dependent Dirichlet processes. Journal of the American Statistical Association 101(473), 179194.
Hall, P., Racine, J., & Li, Q. (2004) Cross-validation and the estimation of conditional probability densities. Journal of the American Statistical Association 99(468), 10151026.
Hayfield, T. & Racine, J.S. (2008) Nonparametric econometrics: The np package. Journal of Statistical Software 27(5), 132.
Huang, T.M. (2004) Convergence rates for posterior distributions and adaptive estimation. The Annals of Statistics 32(4), 15561593.
Jacobs, R.A., Jordan, M.I., Nowlan, S.J., & Hinton, G.E. (1991) Adaptive mixtures of local experts. Neural Computation 3(1), 7987.
Jordan, M. & Xu, L. (1995) Convergence results for the em approach to mixtures of experts architectures. Neural Networks 8(9), 14091431.
Keane, M. & Stavrunova, O. (2011) A smooth mixture of tobits model for healthcare expenditure. Health Economics 20(9), 11261153.
Kruijer, W., Rousseau, J., & van der Vaart, A. (2010) Adaptive Bayesian density estimation with location-scale mixtures. Electronic Journal of Statistics 4, 12251257.
Li, F., Villani, M., & Kohn, R. (2010) Flexible modeling of conditional distributions using smooth mixtures of asymmetric student t densities. Journal of Statistical Planning and Inference 140(12), 36383654.
Li, Q. & Racine, J.S. (2007) Nonparametric Econometrics: Theory and Practice. Princeton University Press.
MacEachern, S.N. (1999) Dependent nonparametric processes. Proceedings of the Section on Bayesian Statistical Science, pp. 5055. American Statistical Association.
Norets, A. (2010) Approximation of conditional densities by smooth mixtures of regressions. The Annals of Statistics 38(3), 17331766.
Norets, A. (2015) Optimal retrospective sampling for a class of variable dimension models. Unpublished manuscript, Brown University.
Norets, A. & Pelenis, J. (2012) Bayesian modeling of joint and conditional distributions. Journal of Econometrics 168, 332346.
Norets, A. & Pelenis, J. (2014) Posterior consistency in conditional density estimation by covariate dependent mixtures. Econometric Theory 30, 606646.
Pati, D., Dunson, D.B., & Tokdar, S.T. (2013) Posterior consistency in conditional distribution estimation. Journal of Multivariate Analysis 116, 456472.
Peng, F., Jacobs, R.A., & Tanner, M.A. (1996) Bayesian inference in mixtures-of-experts and hierarchical mixtures-of-experts models with an application to speech recognition. Journal of the American Statistical Association 91(435), 953960.
Rousseau, J. (2010) Rates of convergence for the posterior distributions of mixtures of betas and adaptive nonparametric estimation of the density. The Annals of Statistics 38(1), 146180.
Scricciolo, C. (2006) Convergence rates for Bayesian density estimation of infinite-dimensional exponential families. Annals of Statatistics 34(6), 28972920.
Shen, W. & Ghosal, S. (2016) Adaptive Bayesian density regression for high-dimensional data. Bernoulli 22(1), 396420.
Shen, W., Tokdar, S.T., & Ghosal, S. (2013) Adaptive Bayesian multivariate density estimation with Dirichlet mixtures. Biometrika 100(3), 623640.
Tokdar, S., Zhu, Y., & Ghosh, J. (2010) Bayesian density regression with logistic Gaussian process and subspace projection. Bayesian Analysis 5(2), 319344.
van der Vaart, A.W. & van Zanten, J.H. (2009) Adaptive Bayesian estimation using a Gaussian random field with inverse gamma bandwidth. The Annals of Statistics 37(5B), 26552675.
Villani, M., Kohn, R., & Giordani, P. (2009) Regression density estimation using smooth adaptive Gaussian mixtures. Journal of Econometrics 153(2), 155173.
Villani, M., Kohn, R., & Nott, D.J. (2012) Generalized smooth finite mixtures. Journal of Econometrics 171(2), 121133.
Wade, S., Dunson, D.B., Petrone, S., & Trippa, L. (2014) Improving prediction from Dirichlet process mixtures via enrichment. The Journal of Machine Learning Research 15(1), 10411071.
Wood, S., Jiang, W., & Tanner, M. (2002) Bayesian mixture of splines for spatially adaptive nonparametric regression. Biometrika 89(3), 513528.
Yang, Y. & Tokdar, S.T. (2015) Minimax-optimal nonparametric regression in high dimensions. The Annals of Statistics 43(2), 652674.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Econometric Theory
  • ISSN: 0266-4666
  • EISSN: 1469-4360
  • URL: /core/journals/econometric-theory
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed