Hostname: page-component-8448b6f56d-c47g7 Total loading time: 0 Render date: 2024-04-23T13:58:14.262Z Has data issue: false hasContentIssue false
  • English
  • Français

POSTERIOR CONSISTENCY IN CONDITIONAL DENSITY ESTIMATION BY COVARIATE DEPENDENT MIXTURES

Published online by Cambridge University Press:  18 November 2013

Andriy Norets  and
Justinas Pelenis
Andriy Norets*
Affiliation:
University of Illinois at Urbana-Champaign
Justinas Pelenis
Affiliation:
Institute for Advanced Studies, Vienna
*
*Address corresponding to Andriy Norets, Department of Economics, University of Illinois, 1407 W. Gregory Drive, Urbana, IL 61801; e-mail: anorets@illinois.edu.
Get access Rights & Permissions [Opens in a new window]

Abstract

This paper considers Bayesian nonparametric estimation of conditional densities by countable mixtures of location-scale densities with covariate dependent mixing probabilities. The mixing probabilities are modeled in two ways. First, we consider finite covariate dependent mixture models, in which the mixing probabilities are proportional to a product of a constant and a kernel and a prior on the number of mixture components is specified. Second, we consider kernel stick-breaking processes for modeling the mixing probabilities. We show that the posterior in these two models is weakly and strongly consistent for a large class of data-generating processes. A simulation study conducted in the paper demonstrates that the models can perform well in small samples.

Type
ARTICLES
Information
Econometric Theory , Volume 30 , Issue 3 , June 2014 , pp. 606 - 646
Copyright
Copyright © Cambridge University Press 2013 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Barron, A. (1988) The Exponential Convergence of Posterior Probabilities with Implications for Bayes Estimators of Density Functions, Working paper, University of Illinois.Google Scholar
Barron, A., Schervish, M.J., & Wasserman, L. (1999) The consistency of posterior distributions in nonparametric problems. Annals of Statistics 27, 536–561.Google Scholar
Billingsley, P. (1999) Convergence of Probability Measures. Wiley-Interscience.Google Scholar
Blackwell, D. & MacQueen, J.B. (1973) Ferguson distributions via polya urn schemes. Annals of Statistics 1, 353–355.Google Scholar
Chen, X. (2007) Large sample sieve estimation of semi-nonparametric models. In Heckman, J. & Leamer, E. (eds), Handbook of Econometrics, vol. 6 of Handbook of Econometrics, ch. 76. Elsevier.Google Scholar
Chung, Y. & Dunson, D.B. (2009) Nonparametric bayes conditional distribution modeling with variable selection. Journal of the American Statistical Association 104, 1646–1660.Google Scholar
Cook, S.R., Gelman, A., & Rubin, D.B. (2006) Validation of software for bayesian models using posterior quantiles. Journal of Computational and Graphical Statistics 15, 675–692.Google Scholar
De Iorio, M., Muller, P., Rosner, G.L., & MacEachern, S.N. (2004) An ANOVA model for dependent random measures. Journal of the American Statistical Association 99, 205–215.Google Scholar
Dey, D., Muller, P., & Sinha, D. (eds.) (1998) Practical Nonparametric and Semiparametric Bayesian Statistics, Lecture Notes in Statistics, Vol. 133. Springer.Google Scholar
DiNardo, J. & Tobias, J.L. (2001) Nonparametric density and regression estimation. Journal of Economic Perspectives 15, 11–28.Google Scholar
Dunson, D.B. & Park, J.-H. (2008) Kernel stick-breaking processes. Biometrika 95, 307–323.Google Scholar
Dunson, D.B., Pillai, N., & Park, J.-H. (2007) Bayesian density regression. Journal of the Royal Statistical Society. Series B (Statistical Methodology) 69, 163–183.Google Scholar
Escobar, M. & West, M. (1995) Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association 90, 577–588.Google Scholar
Escobar, M.D. (1994) Estimating normal means with a dirichlet process prior. Journal of the American Statistical Association 89, 268–277.Google Scholar
Fan, J. (2005) A selective overview of nonparametric methods in financial econometrics. Statistical Science 20, 317–337.Google Scholar
Ferguson, T.S. (1973) A Bayesian analysis of some nonparametric problems, Annals of Statistics 1, 209–230.Google Scholar
Genovese, C.R. & Wasserman, L. (2000) Rates of convergence for the gaussian mixture sieve. Annals of Statistics 28, 1105–1127.Google Scholar
Geweke, J. (2004) Getting it right: joint distribution tests of posterior simulators. Journal of the American Statistical Association 99, 799–804.Google Scholar
Geweke, J. (2005) Contemporary Bayesian Econometrics and Statistics. Wiley-Interscience.Google Scholar
Geweke, J. & Keane, M. (2007) Smoothly mixing regressions. Journal of Econometrics 138, 252–290.Google Scholar
Ghosal, S., Ghosh, J.K., & Ramamoorthi, R.V. (1999) Posterior consistency of dirichlet mixtures in density estimation. Annals of Statistics 27, 143–158.Google Scholar
Ghosal, S., Ghosh, J.K., & ven der Vaart, A.W. (2000) Convergence rates of posterior distributions. Annals of Statistics 28, 500–531.Google Scholar
Ghosal, S. & Tang, Y. (2006) Bayesian consistency for Markov processes. Sankhya 68, 227–239.Google Scholar
Ghosh, J. & Ramamoorthi, R. (2003) Bayesian Nonparametrics. 1st ed. Springer.Google Scholar
Green, P.J. (1995) Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82, 711–732.Google Scholar
Griffin, J.E. & Steel, M.F.J. (2006) Order-based dependent Dirichlet processes. Journal of the American Statistical Association 101, 179–194.Google Scholar
Hall, P., Racine, J., & Li, Q. (2004) Cross-validation and the estimation of conditional probability densities. Journal of the American Statistical Association 99, 1015–1026.Google Scholar
Hall, P., Wolff, R.C.L., & Yao, Q. (1999) Methods for estimating a conditional distribution function. Journal of the American Statistical Association 94, 154–163.Google Scholar
Hayfield, T. & Racine, J.S. (2008) Nonparametric econometrics: The np package. Journal of Statistical Software 27, 1–32.Google Scholar
Ichimura, H. & Todd, P.E. (2007) Implementing nonparametric and semiparametric estimators. In Handbook of Econometrics, vol. 6, Part B, pp. 5369–5468. Elsevier.Google Scholar
Jacobs, R.A., Jordan, M.I., Nowlan, S.J., & Hinton, G.E. (1991) Adaptive mixtures of local experts. Neural Computation 3, 79–87.Google Scholar
Jambunathan, M.V. (1954) Some properties of beta and gamma distributions. Annals of Mathematical Statistics 25, 401–405.Google Scholar
Jordan, M. & Xu, L. (1995) Convergence results for the EM approach to mixtures of experts architectures. Neural Networks 8, 1409–1431.Google Scholar
Koenker, R. & Hallock, K.F. (2001) Quantile regression. Journal of Economic Perspectives 15, 143–156.Google Scholar
Kruijer, W., Rousseau, J., & van der Vaart, A. (2010) Adaptive Bayesian density estimation with location-scale mixtures. Electronic Journal of Statistics 4, 1225–1257.Google Scholar
Li, F., Villani, M., & Kohn, R. (2010) Flexible modeling of conditional distributions using smooth mixtures of asymmetric student t densities. Journal of Statistical Planning and Inference 140, 3638–3654.Google Scholar
Li, J.Q. & Barron, A.R. (1999) Mixture density estimation. In Advances in Neural Information Processing Systems 12, pp. 279–285. MIT Press.Google Scholar
Liang, S., Carlin, B.P., & Gelfand, A.E. (2009) Analysis of Minnesota colon and rectum cancer point patterns with spatial and nonspatial covariate information. Annals of Applied Statistics 3, 943–962.Google Scholar
MacEachern, S.N. (1999) Dependent nonparametric processes. ASA Proceedings of the Section on Bayesian Statistical Science.Google Scholar
McLachlan, G. & Peel, D. (2000) Finite Mixture Models. Wiley.Google Scholar
Muller, P., Erkanli, A., & West, M. (1996) Bayesian curve fitting using multivariate normal mixtures. Biometrika 83, 67–79.Google Scholar
Neal, R.M. (2003) Slice sampling. Annals of Statistics 31, 705–767 , with discussions and a rejoinder by the author.Google Scholar
Norets, A. (2010) Approximation of conditional densities by smooth mixtures of regressions. Annals of Statistics 38, 1733–1766.Google Scholar
Norets, A. & Pelenis, J. (2012) Bayesian modeling of joint and conditional distributions. Journal of Econometrics 168, 332–346.Google Scholar
Papaspiliopoulos, O. (2008) A note on Posterior sampling from Dirichlet mixture models. Preprint.Google Scholar
Papaspiliopoulos, O. & Roberts, G.O. (2008) Retrospective Markov chain Monte Carlo methods for Dirichlet process hierarchical models. Biometrika 95, 169–186.Google Scholar
Pati, D., Dunson, D.B., & Tokdar, S.T. (2013) Posterior consistency in conditional distribution estimation. Journal of Multivariate Analysis 116, 456–472.Google Scholar
Peng, F., Jacobs, R.A., & Tanner, M.A. (1996) Bayesian inference in mixtures-of-experts and hierarchical mixtures-of-experts models with an application to speech recognition. Journal of the American Statistical Association 91, 953–960.Google Scholar
Roeder, K. & Wasserman, L. (1997) Practical Bayesian density estimation using mixtures of normals. Journal of the American Statistical Association 92, 894–902.Google Scholar
Rousseau, J. (2010) Rates of convergence for the posterior distributions of mixtures of betas and adaptive nonparametric estimation of the density. Annals of Statistics 38, 146–180.Google Scholar
Schwartz, L. (1965) On Bayes procedures. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete 4, 10–26.Google Scholar
Sethuraman, J. (1994) A constructive definition of Dirichlet priors. Statistica Sinica 4, 639–650.Google Scholar
Taddy, M.A. & Kottas, A. (2010) A Bayesian nonparametric approach to inference for quantile regression. Journal of Business and Economic Statistics 28, 357–369.Google Scholar
Tokdar, S. (2007) Towards a faster implementation of density estimation with logistic Gaussian process priors. Journal of Computational and Graphical Statistics 16, 633–655.Google Scholar
Tokdar, S., Zhu, Y., & Ghosh, J. (2010) Bayesian density regression with logistic Gaussian process and subspace projection. Bayesian Analysis 5, 319–344.Google Scholar
Tokdar, S.T. (2006) Posterior consistency of Dirichlet location-scale mixture of normals in density estimation and regression. Sankhya: The Indian Journal of Statistics 67, 99–100.Google Scholar
Tokdar, S.T. & Ghosh, J.K. (2007) Posterior consistency of logistic Gaussian process priors in density estimation. Journal of Statistical Planning and Inference 137, 34–42.Google Scholar
Tran, M.-N., Nott, D.J., & Kohn, R. (2012) Simultaneous variable selection and component selection for regression density estimation with mixtures of heteroscedastic experts. Electronic Journal of Statistics 6, 1170–1199.Google Scholar
van der Vaart, A.W. & van Zanten, J.H. (2008) Rates of contraction of posterior distributions based on Gaussian process priors. Annals of Statistics 36, 1435–1463.Google Scholar
van der Vaart, A.W. & van Zanten, J.H. (2009) Adaptive Bayesian estimation using a Gaussian random field with inverse gamma bandwidth. Annals of Statistics 37, 2655–2675.Google Scholar
Villani, M., Kohn, R., & Giordani, P. (2009) Regression density estimation using smooth adaptive Gaussian mixtures. Journal of Econometrics 153, 155–173.Google Scholar
Villani, M., Kohn, R., & Nott, D.J. (2012) Generalized smooth finite mixtures. Journal of Econometrics 171, 121–133.Google Scholar
Walker, S.G. (2004) Posterior consistency of Dirichlet mixtures in density estimation. Annals of Statistics 32, 2028–2043.Google Scholar
Walker, S.G. (2007) Sampling the Dirichlet mixture model with slices. Communications in Statistics - Simulation and Computation 36, 45–54.Google Scholar
Wood, S., Jiang, W., & Tanner, M. (2002) Bayesian mixture of splines for spatially adaptive nonparametric regression. Biometrika 89, 513–528.Google Scholar
Wu, Y. & Ghosal, S. (2010) The L1-consistency of Dirichlet mixtures in multivariate Bayesian density estimation. Journal of Multivariate Analysis 101, 2411–2419.Google Scholar
Yatchew, A. (1998) Nonparametric regression techniques in economics. Journal of Economic Literature 36, 669–721.Google Scholar
Zeevi, A.J. & Meir, R. (1997) Density estimation through convex combinations of densities: approximation and estimation bounds. Neural Networks 10, 99–109.Google Scholar