Skip to main content


  • Andriy Norets (a1) and Justinas Pelenis (a2)

This paper considers Bayesian nonparametric estimation of conditional densities by countable mixtures of location-scale densities with covariate dependent mixing probabilities. The mixing probabilities are modeled in two ways. First, we consider finite covariate dependent mixture models, in which the mixing probabilities are proportional to a product of a constant and a kernel and a prior on the number of mixture components is specified. Second, we consider kernel stick-breaking processes for modeling the mixing probabilities. We show that the posterior in these two models is weakly and strongly consistent for a large class of data-generating processes. A simulation study conducted in the paper demonstrates that the models can perform well in small samples.

Corresponding author
*Address corresponding to Andriy Norets, Department of Economics, University of Illinois, 1407 W. Gregory Drive, Urbana, IL 61801; e-mail:
Hide All
Barron, A. (1988) The Exponential Convergence of Posterior Probabilities with Implications for Bayes Estimators of Density Functions, Working paper, University of Illinois.
Barron, A., Schervish, M.J., & Wasserman, L. (1999) The consistency of posterior distributions in nonparametric problems. Annals of Statistics 27, 536561.
Billingsley, P. (1999) Convergence of Probability Measures. Wiley-Interscience.
Blackwell, D. & MacQueen, J.B. (1973) Ferguson distributions via polya urn schemes. Annals of Statistics 1, 353355.
Chen, X. (2007) Large sample sieve estimation of semi-nonparametric models. In Heckman, J. & Leamer, E. (eds), Handbook of Econometrics, vol. 6 of Handbook of Econometrics, ch. 76. Elsevier.
Chung, Y. & Dunson, D.B. (2009) Nonparametric bayes conditional distribution modeling with variable selection. Journal of the American Statistical Association 104, 16461660.
Cook, S.R., Gelman, A., & Rubin, D.B. (2006) Validation of software for bayesian models using posterior quantiles. Journal of Computational and Graphical Statistics 15, 675692.
De Iorio, M., Muller, P., Rosner, G.L., & MacEachern, S.N. (2004) An ANOVA model for dependent random measures. Journal of the American Statistical Association 99, 205215.
Dey, D., Muller, P., & Sinha, D. (eds.) (1998) Practical Nonparametric and Semiparametric Bayesian Statistics, Lecture Notes in Statistics, Vol. 133. Springer.
DiNardo, J. & Tobias, J.L. (2001) Nonparametric density and regression estimation. Journal of Economic Perspectives 15, 1128.
Dunson, D.B. & Park, J.-H. (2008) Kernel stick-breaking processes. Biometrika 95, 307323.
Dunson, D.B., Pillai, N., & Park, J.-H. (2007) Bayesian density regression. Journal of the Royal Statistical Society. Series B (Statistical Methodology) 69, 163183.
Escobar, M. & West, M. (1995) Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association 90, 577588.
Escobar, M.D. (1994) Estimating normal means with a dirichlet process prior. Journal of the American Statistical Association 89, 268277.
Fan, J. (2005) A selective overview of nonparametric methods in financial econometrics. Statistical Science 20, 317337.
Ferguson, T.S. (1973) A Bayesian analysis of some nonparametric problems, Annals of Statistics 1, 209230.
Genovese, C.R. & Wasserman, L. (2000) Rates of convergence for the gaussian mixture sieve. Annals of Statistics 28, 11051127.
Geweke, J. (2004) Getting it right: joint distribution tests of posterior simulators. Journal of the American Statistical Association 99, 799804.
Geweke, J. (2005) Contemporary Bayesian Econometrics and Statistics. Wiley-Interscience.
Geweke, J. & Keane, M. (2007) Smoothly mixing regressions. Journal of Econometrics 138, 252290.
Ghosal, S., Ghosh, J.K., & Ramamoorthi, R.V. (1999) Posterior consistency of dirichlet mixtures in density estimation. Annals of Statistics 27, 143158.
Ghosal, S., Ghosh, J.K., & ven der Vaart, A.W. (2000) Convergence rates of posterior distributions. Annals of Statistics 28, 500531.
Ghosal, S. & Tang, Y. (2006) Bayesian consistency for Markov processes. Sankhya 68, 227239.
Ghosh, J. & Ramamoorthi, R. (2003) Bayesian Nonparametrics. 1st ed. Springer.
Green, P.J. (1995) Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82, 711732.
Griffin, J.E. & Steel, M.F.J. (2006) Order-based dependent Dirichlet processes. Journal of the American Statistical Association 101, 179194.
Hall, P., Racine, J., & Li, Q. (2004) Cross-validation and the estimation of conditional probability densities. Journal of the American Statistical Association 99, 10151026.
Hall, P., Wolff, R.C.L., & Yao, Q. (1999) Methods for estimating a conditional distribution function. Journal of the American Statistical Association 94, 154163.
Hayfield, T. & Racine, J.S. (2008) Nonparametric econometrics: The np package. Journal of Statistical Software 27, 132.
Ichimura, H. & Todd, P.E. (2007) Implementing nonparametric and semiparametric estimators. In Handbook of Econometrics, vol. 6, Part B, pp. 53695468. Elsevier.
Jacobs, R.A., Jordan, M.I., Nowlan, S.J., & Hinton, G.E. (1991) Adaptive mixtures of local experts. Neural Computation 3, 7987.
Jambunathan, M.V. (1954) Some properties of beta and gamma distributions. Annals of Mathematical Statistics 25, 401405.
Jordan, M. & Xu, L. (1995) Convergence results for the EM approach to mixtures of experts architectures. Neural Networks 8, 14091431.
Koenker, R. & Hallock, K.F. (2001) Quantile regression. Journal of Economic Perspectives 15, 143156.
Kruijer, W., Rousseau, J., & van der Vaart, A. (2010) Adaptive Bayesian density estimation with location-scale mixtures. Electronic Journal of Statistics 4, 12251257.
Li, F., Villani, M., & Kohn, R. (2010) Flexible modeling of conditional distributions using smooth mixtures of asymmetric student t densities. Journal of Statistical Planning and Inference 140, 36383654.
Li, J.Q. & Barron, A.R. (1999) Mixture density estimation. In Advances in Neural Information Processing Systems 12, pp. 279285. MIT Press.
Liang, S., Carlin, B.P., & Gelfand, A.E. (2009) Analysis of Minnesota colon and rectum cancer point patterns with spatial and nonspatial covariate information. Annals of Applied Statistics 3, 943962.
MacEachern, S.N. (1999) Dependent nonparametric processes. ASA Proceedings of the Section on Bayesian Statistical Science.
McLachlan, G. & Peel, D. (2000) Finite Mixture Models. Wiley.
Muller, P., Erkanli, A., & West, M. (1996) Bayesian curve fitting using multivariate normal mixtures. Biometrika 83, 6779.
Neal, R.M. (2003) Slice sampling. Annals of Statistics 31, 705767 , with discussions and a rejoinder by the author.
Norets, A. (2010) Approximation of conditional densities by smooth mixtures of regressions. Annals of Statistics 38, 17331766.
Norets, A. & Pelenis, J. (2012) Bayesian modeling of joint and conditional distributions. Journal of Econometrics 168, 332346.
Papaspiliopoulos, O. (2008) A note on Posterior sampling from Dirichlet mixture models. Preprint.
Papaspiliopoulos, O. & Roberts, G.O. (2008) Retrospective Markov chain Monte Carlo methods for Dirichlet process hierarchical models. Biometrika 95, 169186.
Pati, D., Dunson, D.B., & Tokdar, S.T. (2013) Posterior consistency in conditional distribution estimation. Journal of Multivariate Analysis 116, 456472.
Peng, F., Jacobs, R.A., & Tanner, M.A. (1996) Bayesian inference in mixtures-of-experts and hierarchical mixtures-of-experts models with an application to speech recognition. Journal of the American Statistical Association 91, 953960.
Roeder, K. & Wasserman, L. (1997) Practical Bayesian density estimation using mixtures of normals. Journal of the American Statistical Association 92, 894902.
Rousseau, J. (2010) Rates of convergence for the posterior distributions of mixtures of betas and adaptive nonparametric estimation of the density. Annals of Statistics 38, 146180.
Schwartz, L. (1965) On Bayes procedures. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete 4, 1026.
Sethuraman, J. (1994) A constructive definition of Dirichlet priors. Statistica Sinica 4, 639650.
Taddy, M.A. & Kottas, A. (2010) A Bayesian nonparametric approach to inference for quantile regression. Journal of Business and Economic Statistics 28, 357369.
Tokdar, S. (2007) Towards a faster implementation of density estimation with logistic Gaussian process priors. Journal of Computational and Graphical Statistics 16, 633655.
Tokdar, S., Zhu, Y., & Ghosh, J. (2010) Bayesian density regression with logistic Gaussian process and subspace projection. Bayesian Analysis 5, 319344.
Tokdar, S.T. (2006) Posterior consistency of Dirichlet location-scale mixture of normals in density estimation and regression. Sankhya: The Indian Journal of Statistics 67, 99100.
Tokdar, S.T. & Ghosh, J.K. (2007) Posterior consistency of logistic Gaussian process priors in density estimation. Journal of Statistical Planning and Inference 137, 3442.
Tran, M.-N., Nott, D.J., & Kohn, R. (2012) Simultaneous variable selection and component selection for regression density estimation with mixtures of heteroscedastic experts. Electronic Journal of Statistics 6, 11701199.
van der Vaart, A.W. & van Zanten, J.H. (2008) Rates of contraction of posterior distributions based on Gaussian process priors. Annals of Statistics 36, 14351463.
van der Vaart, A.W. & van Zanten, J.H. (2009) Adaptive Bayesian estimation using a Gaussian random field with inverse gamma bandwidth. Annals of Statistics 37, 26552675.
Villani, M., Kohn, R., & Giordani, P. (2009) Regression density estimation using smooth adaptive Gaussian mixtures. Journal of Econometrics 153, 155173.
Villani, M., Kohn, R., & Nott, D.J. (2012) Generalized smooth finite mixtures. Journal of Econometrics 171, 121133.
Walker, S.G. (2004) Posterior consistency of Dirichlet mixtures in density estimation. Annals of Statistics 32, 20282043.
Walker, S.G. (2007) Sampling the Dirichlet mixture model with slices. Communications in Statistics - Simulation and Computation 36, 4554.
Wood, S., Jiang, W., & Tanner, M. (2002) Bayesian mixture of splines for spatially adaptive nonparametric regression. Biometrika 89, 513528.
Wu, Y. & Ghosal, S. (2010) The L1-consistency of Dirichlet mixtures in multivariate Bayesian density estimation. Journal of Multivariate Analysis 101, 24112419.
Yatchew, A. (1998) Nonparametric regression techniques in economics. Journal of Economic Literature 36, 669721.
Zeevi, A.J. & Meir, R. (1997) Density estimation through convex combinations of densities: approximation and estimation bounds. Neural Networks 10, 99109.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Econometric Theory
  • ISSN: 0266-4666
  • EISSN: 1469-4360
  • URL: /core/journals/econometric-theory
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed